[Catalyst] Requirements for Catalyst

[Joel Bernstein]

2009/2/25 Bill Moseley <moseley at hank.org>:
> On Wed, Feb 25, 2009 at 08:01:55AM -0800, J. Shirley wrote:
>> As other people mentioned, you likely have a memory leak or some other
>> problem.  I have a much larger application than yours, and my memory usage
>> doesn't grow.
>>
>> Please don't spread FUD saying that Catalyst "literally explodes".  It does
>> if you poorly write your application and don't check for memory leaks.
>
> Oh good.  This reminded me I was looking for a leak a few months back.
> It wasn't significant enough to worry about at the time (thanks to our
> memory-fat servers).
>
> I was playing with Linux::Smaps and trying to understand why memory
> was going up.  Again, I never had time to look at it in detail, and it's
> probably something obvious or simply just an invalid test.  But, my
> script is really simple.
>
> What was curious was a considerable large change in memory after the
> first request, but memory still increases every request.

The large change in memory after first request seems inevitable.
Presumably your Catalyst processes are forked from a common parent?
They will share the parent process's memory with copy-on-write
semantics. As soon as your app serves requests, it modifies data in
memory. Any pages containing storage for those shared variables will
be copied (i.e. allocated within the current process) and written.
Thus the first hit is likely to show the process growing.

On the other hand, the memory leak you describe seems odd.

>
> $ perl /home/moseley/catalyst_leak.pl
> ping
> Initial rss is  7488kb
> Current rss is  9648kb after 500 requests
> Current rss is  9676kb after 1000 requests
> Current rss is  9708kb after 1500 requests
> Current rss is  9736kb after 2000 requests
> Current rss is  9768kb after 2500 requests
> Current rss is  9796kb after 3000 requests
> Current rss is  9828kb after 3500 requests
> Current rss is  9852kb after 4000 requests
> Current rss is  9880kb after 4500 requests
> Current rss is  9912kb after 5000 requests
> Current rss is  9940kb after 5500 requests
> Current rss is  9976kb after 6000 requests
> Current rss is 10004kb after 6500 requests
> Current rss is 10036kb after 7000 requests
> Current rss is 10064kb after 7500 requests
> Current rss is 10096kb after 8000 requests
> Current rss is 10124kb after 8500 requests
> Current rss is 10156kb after 9000 requests
> Current rss is 10184kb after 9500 requests
> Current rss is 10212kb after 10000 requests

So you're leaking about 57 bytes per request. I assume you're on a
32-bit machine? Sounds like a single large scalar leak. Time to pull
out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
Devel::Gladiator, etc) and start searching your app.

>
> $ cat catalyst_leak.pl
> use strict;
> use warnings;
> use HTTP::Request::AsCGI;
> use Catalyst::Utils;
> use Linux::Smaps;
> my $smap = Linux::Smaps->new( $$ );
> App->setup;
>
> my $r = make_request();  # prime the pump
> print $r->content, "\n";
>
> printf "Initial rss is %5dkb\n", $smap->rss;
>
> for my $count ( 1 .. 10_000 ) {
>    make_request();
>    next if $count % 500;
>
>    $smap->update;
>    printf( "Current rss is %5dkb after %d requests\n", $smap->rss, $count );
> }
>
>
> sub make_request {
>    my $request = Catalyst::Utils::request( '/ping' );
>    my $cgi     = HTTP::Request::AsCGI->new( $request, %ENV )->setup;
>    App->handle_request;
>    return $cgi->restore->response;
> }

This seems odd to me. Where did you find this pattern? I've never seen
anybody test a Catalyst webapp in this way before.
Are you confident that the CGI-request lib doesn't leak, for example?
Can you reproduce this memory leakage using a realistic test, against
the running web app, over HTTP?

> package App;
> use strict;
> use warnings;
> use Catalyst;
>
> sub ping : Local {
>    my ( $self, $c ) = @_;
>    $c->res->body( 'ping' );
> }

Admittedly this looks like an awfully simple app, there doesn't seem
an obvious leak here. Hence we wondering if your rather odd-looking
test is to blame.

What versions of Catalyst, any plugins, etc? Can you provide more data
to support this?

/joel