[Catalyst] Production session issue - commercial support inquiry?

[Matt Pitts]

We had another major incident of session cross-over this morning which
happened right after I pushed out a small change and reload/restarted
the instances.

I personally experienced the session cross-over twice. In each instance
I captured my session id and went to compare it to a dump of all
sessions and found that no other session had the same data as mine.
However, the 2 session id's *were* the same and I had cleared my cookies
between these two instances.

So, I could be getting dups out of generate_session_id, but I'm also
wondering if it's not some weird process issue where the same session ID
is not being generated twice, but is just getting shoved into the
response headers of 2 different requests. Anyone know if this is
plausible?

If P::Session is generating duplicate IDs I doubt I'd be the only one
having the issue. BTW, the cat app only averages about 12 req/s and this
issue doesn't seem to be very dependent on traffic.

Along the lines of the process question...

What, if any, are the ramifications of running two instances of the app
(one HTTP::Prefork and one FCGI) out of the same "root". In my
particular case the FCGI instance is for talking to a local Apache
instance that is providing SSL access and the HTTP::Prefork instance is
to provide direct HTTP access to areas where SSL isn't needed. I wasn't
originally worried about this because they're separate process, but
right now I'm looking at anything that might be causing my problems.
Could a situation like this cause strange behavior in regards to session
ids.

Right now I have a set of changes primed to push out that completely
removed P::Session and uses P::CookiedSession instead. I guess if I
don't get a resolution to the issues with P::Session by Monday I'll be
pushing this out and then I'll know it's not P::Session problem if it
continues.

TIA

-matt

> -----Original Message-----
> From: Matt Pitts [mailto:mpitts at a3its.com]
> Sent: Thursday, January 08, 2009 10:20 AM
> To: The elegant MVC web framework
> Subject: [Catalyst] Production session issue - commercial support
> inquiry?
> 
> A Cat app for one of our clients has been experiencing random session
> cross-over issues for the past several months. The first reported
> instance was on Oct. 16 and I have spent countless hours looking at
any
> changes that took place around that time to try to isolate the problem
> without success.
> 
> Here is an overview of the application setup:
> 
>  - Catalyst 5.70014
>  - Catalyst::Plugin::Session 0.20
>  - single Apache 2.2 load balancer front end handling remote SSL and
> non-SSL requests
>  - two backend servers running Cat app via HTTP::Prefork for non-SSL
> traffic
>  - each backend also runs Apache/mod_ssl + mod_fastcgi in order to
> provide an SSL channel to the backends
>  - session storage via DBIC (was originally memcached)
> 
> The symptoms of the session issues:
> 
>  - users "seeing" other user's items in their Cart
>  - "club" members updating their information, but the update gets tied
> to another member
> 
> I have personally experienced the issue myself while the site was
under
> a bit of traffic - I added an item to my Cart and I had other items in
> there that I didn't touch. I grabbed my cookie and tracked down my
> session and it was pointing to a cart_id that had all the items in it
> that were showing on the page. The only conclusion I can draw from
this
> is that two separate sessions somehow got mapped to the same cart_id -
> which I don't know where this could be happening at. I'm fairly
> confident that it's not my Cart code because the issue is also
> happening
> on another part of the site - a member's area.
> 
> In an effort to track down the issue I have done the following (not
> necessary in chronological order):
> 
>  - put checks in place during Cart retrieval to help ensure a good
> "load"
> 	Result: no change
> 
>  - applied both "finalize_race_condition" and "flash_in_stash" patches
> from Sergio Salvi
> 
>
(http://dev.catalyst.perl.org/svnweb/Catalyst/browse/branches/Catalyst-
> P
> lugin-Session/both)
>  	Result: no change
> 
>  - taken one of the backends out of the pool for several days
> 	Result: no change
> 
>  - switched session storage to FastMmap while using only a single
> backend
> 	Result: no change
> 
>  - switched session storage to DBIC
> 	Result: no change
> 
>  - updated C::P::Session from 0.19 to 0.20
> 	Result: no change
> 
>  - changed deployment model from FastCGI to HTTP::Prefork and back
> 	Result: no change
> 
>  - setup SHH tunnel for SSL traffic to go directly to HTTP::Prefork
> backend instances (completely eliminate FastCGI components)
> 	Result: no change
> 
>  - attempted to replicate problem in a test environment using a
forking
> site crawler I hacked up
>  	Result: no success
> 
> All in all, I'm fairly confident that this issue is either an error in
> my code/setup of Catalyst and C::P::Session or it's a bug somewhere in
> Catalyst or C::P::Session. I know there's been some traffic on the
list
> about session issues and some talk of refactoring the session code,
but
> I haven't seen anything that is relevant to this problem.
> 
> Any obvious help or suggestions are *greatly* appreciated, but...
> 
> I'm to the point that I need additional help with this problem and
> since
> I'm the only Linux/Perl guys in our shop I'm calling out. My company
is
> committed to getting this problem resolved and we want to get an idea
> of
> what commercial support is available. Our preference would be on-site
> support - we are located in Greensboro, NC, USA - but we'll we can
work
> with remote support if needed. If you're currently providing
commercial
> support specifically for Catalyst applications and you're interested
in
> the work, please feel free to contact me privately with your
> information
> and rates.
> 
> Much thanks and appreciation,
> 
> Matthew Pitts
> Senior Engineer
> Software and Linux Solutions
> A3 IT Solutions, LLC
> [c] 336.202.3913
> [o] 336.389.1101 ex117
> [e] mpitts at a3its.com
> 
> 
> 
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-
> archive.com/catalyst at lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/