[Catalyst] Distributed session storage problems/questions

[Matt Pitts]

We have a load balanced application where the proxy does *not* force any
type of sticky session with each backend. The app was running in
production as a single instance (pre load-balanced) and we moved it
behind the proxy to be able to add backends and make it load balanced. I
had the app in test for several weeks as a load-balanced setup using
memcached to store sessions across the 2 backends and I am pretty
confident that the auth areas that relied on session storage were
working fine.

 

As of last night the app throws errors (in UAT) about every other
request when inside an auth area with the 2 backends enabled. I can't
get the actual message as this time, but it basically was a "Can't use
12345 as a HASHREF..." type of error where '12345' is the ID of the user
I was authenticated as. If I disable one of the backends, clear my
cookies and re-login the entire session works fine. As soon as I
re-enable the other backend I start getting the errors again and it
doesn't matter which backend I use first, they both work independent of
one another. Obviously, the first backend that stores the session works
and the other doesn't, but what I don't know is why. 

 

As I stated above, I'm pretty confident that all the auth areas of the
site were functioning in the load-balanced setup before we went to PROD.
So, I looked at what changed and the only thing that changed related to
session storage was that I added a 2nd memcached backend instance to the
config file. I wondered if this might have been part of the problem, but
I have since dropped back to only one memcached instance, restarted it
and the errors still persist.

 

Here is the relevant config section:

 

session:

  flash_to_stash: 1

  expires: 3600

  memcached_new_args:

    data:

      - 10.xxx.xxx.xxx:11211

    namespace: tangeroutlet_tangerweb_session_uat

 

In an effort to troubleshoot this I have also tried to implement
memcached storage using C::P::Session::Store::Cache and
C::P::Cache::Memcached, but I haven't gotten this to work yet.

 

I'm hoping that answers to the following questions will guide me to a
solution:

 

1. What is the acceptable way to use memcached for session storage? Is
it using C::P::Session::Store::Memcached or is it
C::P::Session::Store::Cache in conjunction with C::P::Cache::Memcached?

 

2. Is memcached even a "good" way to store sessions? DBIC seems to be a
popular option, but I figured memcached's considerable speed would make
it the preferred choice.

 

3. When using memcached to store sessions, is it OK to have more than
one backend? At first I didn't think anything about adding another
backend, but then I thought that the 2 memcached instances wouldn't have
the same set of data which would cause problems with sessions, unless of
course the session writes get distributed to all the backends. I've
looked into the source of C::P::Session::Store::Memcached,
Cache::Memcached::Managed and Cache::Memcached and cannot tell if the
writes are spread across all backends. Does anyone know the answer to
this?

 

4. Is it just crazy to run a load balanced setup without some type of
sticky session setup on the proxy? If so, any implementations of this
using Apache 2.x mod_proxy(_balancer) as the frontend would be greatly
appreciated.

 

At this point I think I'd be willing to take a performance hit and store
sessions in DBIC if it's going to work better in a distributed
environment.

 

TIA for any input.

 

v/r

 

-matt pitts

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20080220/2564fe56/attachment.htm