[Catalyst] Requirements for Catalyst

[Bill Moseley]

On Wed, Feb 25, 2009 at 05:47:15PM +0000, Joel Bernstein wrote:
> 
> The large change in memory after first request seems inevitable.

I'm not sure it's inevitable, but my assumption is that the first
request allocates some memory that is then reused for subsequent
requests.  The only thing to support that is that it only happens on
the first request.  I frankly don't see where that could be happening
in the Catalyst code, but might have missed something.

Granted "rss" memory usage does count in the end, but it's probably
not a very good way to tell if *Perl* is leaking memory.  I'm not an
expert in the details of memory allocation for a Linux process.

Were you able to reproduce the same results on your machine with this
script?


> Presumably your Catalyst processes are forked from a common parent?
> They will share the parent process's memory with copy-on-write
> semantics.

Yes, the web app is loaded in the parent, but no request is done in
the parent, so this loss of memory would be per-child.  That is, this
first chunk of memory is after calling handle_request the first time
and that happens in the child.

> On the other hand, the memory leak you describe seems odd.

That's what I thought.  I tried a few different leak modules but I
either got a *ton* of reported SVs or none -- and I ran out of
time to try and figure it out.


> So you're leaking about 57 bytes per request. I assume you're on a
> 32-bit machine? Sounds like a single large scalar leak. Time to pull
> out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
> Devel::Gladiator, etc) and start searching your app.

I tried Devel::Leak and Devel::Cycle but could not identify.  I'll try
again some day when I have more time.


> > sub make_request {
> >    my $request = Catalyst::Utils::request( '/ping' );
> >    my $cgi     = HTTP::Request::AsCGI->new( $request, %ENV )->setup;
> >    App->handle_request;
> >    return $cgi->restore->response;
> > }
> 
> This seems odd to me. Where did you find this pattern? I've never seen
> anybody test a Catalyst webapp in this way before.

It's not a web app at this point -- just a Catalyst app. ;)  It's just
faking up a request and passing it to the app's handle_request method
just like an Engine would do.

> Are you confident that the CGI-request lib doesn't leak, for example?
> Can you reproduce this memory leakage using a realistic test, against
> the running web app, over HTTP?

I'm not sure the mock request doesn't leak, true.  But it doesn't
account for the initial big chunk of memory.

The point was to have the smallest amount of code in the test and to
avoid the layers of an Engine or test server -- clearly that would
just add more code to try and figure out where the issue was.

I wrote this test because I noticed that processes grew large over
time.  Limiting requests/child has kept me out of swap, though.

> Admittedly this looks like an awfully simple app, there doesn't seem
> an obvious leak here. Hence we wondering if your rather odd-looking
> test is to blame.
> 
> What versions of Catalyst, any plugins, etc? Can you provide more data
> to support this?

our $VERSION = '5.7015';

There's no plugins, of course.  The entire "app" was shown in the code
I posted.

-- 
Bill Moseley
moseley at hank.org
Sent from my iMutt