[Catalyst] Production session issue - commercial support inquiry?

[Matt Pitts]

I greatly appreciate everyone's input to this problem. Due to workload,
I have not yet had an opportunity to follow up with all the suggestions,
but I plan to if I am not able to resolve this problem by other means.

I do have a follow-up question about another possible cause of the
problem. I realized tonight after looking over some of my @INC
bootstrapping code that for some time the code was actually use(ing)
MyApp *before* the "require MyApp" line in myapp_server.pl. Obviously,
an unnecessary step, but I started wondering if it might be the cause of
the problem, especially after I looked back over my SVN history and
determined that the "use" call went into production about 10 days before
our 1st report of a crossed session. Although, thinking back to when the
first report came through our production engine was FastCGI, which
doesn't appear to have ever had the 2 require(use) calls.

I know that CORE::require isn't *supposed* to load a module if it's
already loaded *and* that Catalyst->setup does a check of
$class->setup_finished to ensure that ->setup isn't called twice, but I
thought I would ask anyway. As an added note I have never once seen the
"Running setup twice is not a good idea." warning in my logs.

Thanks again,
Matt Pitts

> -----Original Message-----
> From: Matt Pitts [mailto:mpitts at a3its.com]
> Sent: Thursday, January 08, 2009 10:20 AM
> To: The elegant MVC web framework
> Subject: [Catalyst] Production session issue - commercial support
> inquiry?
> 
> A Cat app for one of our clients has been experiencing random session
> cross-over issues for the past several months. The first reported
> instance was on Oct. 16 and I have spent countless hours looking at
any
> changes that took place around that time to try to isolate the problem
> without success.
> 
> Here is an overview of the application setup:
> 
>  - Catalyst 5.70014
>  - Catalyst::Plugin::Session 0.20
>  - single Apache 2.2 load balancer front end handling remote SSL and
> non-SSL requests
>  - two backend servers running Cat app via HTTP::Prefork for non-SSL
> traffic
>  - each backend also runs Apache/mod_ssl + mod_fastcgi in order to
> provide an SSL channel to the backends
>  - session storage via DBIC (was originally memcached)
> 
> The symptoms of the session issues:
> 
>  - users "seeing" other user's items in their Cart
>  - "club" members updating their information, but the update gets tied
> to another member
> 
> I have personally experienced the issue myself while the site was
under
> a bit of traffic - I added an item to my Cart and I had other items in
> there that I didn't touch. I grabbed my cookie and tracked down my
> session and it was pointing to a cart_id that had all the items in it
> that were showing on the page. The only conclusion I can draw from
this
> is that two separate sessions somehow got mapped to the same cart_id -
> which I don't know where this could be happening at. I'm fairly
> confident that it's not my Cart code because the issue is also
> happening
> on another part of the site - a member's area.
> 
> In an effort to track down the issue I have done the following (not
> necessary in chronological order):
> 
>  - put checks in place during Cart retrieval to help ensure a good
> "load"
> 	Result: no change
> 
>  - applied both "finalize_race_condition" and "flash_in_stash" patches
> from Sergio Salvi
> 
>
(http://dev.catalyst.perl.org/svnweb/Catalyst/browse/branches/Catalyst-
> P
> lugin-Session/both)
>  	Result: no change
> 
>  - taken one of the backends out of the pool for several days
> 	Result: no change
> 
>  - switched session storage to FastMmap while using only a single
> backend
> 	Result: no change
> 
>  - switched session storage to DBIC
> 	Result: no change
> 
>  - updated C::P::Session from 0.19 to 0.20
> 	Result: no change
> 
>  - changed deployment model from FastCGI to HTTP::Prefork and back
> 	Result: no change
> 
>  - setup SHH tunnel for SSL traffic to go directly to HTTP::Prefork
> backend instances (completely eliminate FastCGI components)
> 	Result: no change
> 
>  - attempted to replicate problem in a test environment using a
forking
> site crawler I hacked up
>  	Result: no success
> 
> All in all, I'm fairly confident that this issue is either an error in
> my code/setup of Catalyst and C::P::Session or it's a bug somewhere in
> Catalyst or C::P::Session. I know there's been some traffic on the
list
> about session issues and some talk of refactoring the session code,
but
> I haven't seen anything that is relevant to this problem.
> 
> Any obvious help or suggestions are *greatly* appreciated, but...
> 
> I'm to the point that I need additional help with this problem and
> since
> I'm the only Linux/Perl guys in our shop I'm calling out. My company
is
> committed to getting this problem resolved and we want to get an idea
> of
> what commercial support is available. Our preference would be on-site
> support - we are located in Greensboro, NC, USA - but we'll we can
work
> with remote support if needed. If you're currently providing
commercial
> support specifically for Catalyst applications and you're interested
in
> the work, please feel free to contact me privately with your
> information
> and rates.
> 
> Much thanks and appreciation,
> 
> Matthew Pitts
> Senior Engineer
> Software and Linux Solutions
> A3 IT Solutions, LLC
> [c] 336.202.3913
> [o] 336.389.1101 ex117
> [e] mpitts at a3its.com
> 
> 
> 
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-
> archive.com/catalyst at lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/