[Catalyst] Memory leak under FastCGI?

[Matt Pitts]

> -----Original Message-----
> From: Matt S Trout [mailto:dbix-class at trout.me.uk]
> Sent: Friday, April 11, 2008 10:35 AM
> To: The elegant MVC web framework
> Subject: Re: [Catalyst] Memory leak under FastCGI?
> 
> On Thu, Apr 10, 2008 at 03:11:56PM -0400, Matt Pitts wrote:
> > > -----Original Message-----
> > > From: Matt S Trout [mailto:dbix-class at trout.me.uk]
> > > Sent: Thursday, April 10, 2008 12:33 PM
> > > To: The elegant MVC web framework
> > > Subject: Re: [Catalyst] Memory leak under FastCGI?
> > >
> > > On Mon, Apr 07, 2008 at 10:52:54AM -0400, Matt Pitts wrote:
> > > > As of now, I'm trying to hack up a better ProcManager based on
> > > > FCGI::Engine::ProcManager that actually recycles its children
> using
> > > > options like MaxRequestPerChild. Hopefully, I'll be able to get
> it
> > > done
> > > > and cleaned up enough to release.
> > >
> > > Please don't; there's work being done on an app plugin that
> implements
> > > this generically, which is a much better approach. Contact John
> >
> > I see "generic" from two directions: Catalyst and FastCGI.
> Implementing
> > this as a subclass of FCGI::Engine::ProcManager makes it available
to
> > anyone using FastCGI and is, therefore, more generic to FastCGI.
> > However, implementing this as a Cat plugin makes it generic (or
> > abstract) to all the C::Engine:: modules, so that all Cat apps can
> enjoy
> > the benefits. I'm not sure that either is the better approach they
> just
> > have different scopes.
> >
> > > Napiorkowski for his current prototype and help get that finished.
> > >
> > > Once you use something like that, the child exits and ProcManager
> just
> > > spawns another one.
> >
> > I don't like this approach because the dying of children isn't being
> > managed in a single place. If it's done like this and the children
> don't
> > coordinate their deaths then you could run into a situation where
> many
> > or all of the children die at the same time - this is bad for
obvious
> > reasons, especially in an external FastCGI app. My plan was do all
> child
> > TERM signaling from within the "manager" app and use some simple
> > heuristics to prevent a mass-child death situation.
> >
> > Obviously, if it's written as a plugin, then I can choose to use or
> not
> > to use - that's be beauty of Catalyst - I guess I'm just trying to
> > defend my position that I like the idea of having a
> MaxRequestsPerChild
> > feature implemented in the "manager" of my FastCGI processes not
> within
> > the children themselves.
> 
> The accept() happens in the child in normal situations though, so the
> parent can't know how many requests a given child has handled without
> additional complication.

Yeap. My version of the "additional complication" is a lite IPC
messaging system using DBM::Deep. I override post_dispatch in the
children to send a message to the queue that says "I just finished
handling a request" and then the manager iterates the message queue
during each daemon loop and keeps track of each child's request count.

This obviously introduces some overhead to the post_dispatch call, but
I'm thinking that since MyApp->handle_request has already been called
that all data has been sent to the client (buffering?), the only thing
that might not have happened yet is the closing of the FastCGI
connection with the socket.

Am I wrong? Can anyone elaborate on the effects of doing this in
post_dispatch?
 
> Basically, you -can't- manage it in a single place. A "don't die for a
> bit,
> somebody else just did" mechanism would be good as a co-ordination
> between
> manager and children and IMO should be implemented separately.

My plan is to build it as a subclass of FCGI::Engine::ProcManager so
folks can use it at their own discretion.

v/r

-matt