[Catalyst] Running Catalyst apps with start_server

[Octavian Rasnita]

Hi t0m,

From: "Tomas Doran" <bobtfish at bobtfish.net>
> 
> On 21 Jan 2012, at 22:16, Octavian Rasnita wrote:
>> But of course, it doesn't go in the background...
>> I can put it to run in the background if I add a " &" after the  
>> command line start_server ... and I've seen that if I do that it  
>> works and I can also add or remove workers with kill -TTIN|TTOU,  
>> but... is it a good way? Or it is better recommended to run starman  
>> directly?
> 
> Any way you like is fine - as we're a ubuntu shop at work, we're  
> mostly deploying things just using upstart to start a (not  
> daemonizing) server - which works well for us.


I intend to use my app under Ubuntu, so I'll probably also want to use upstart, but for the moment I am testing it and not with very good results.

> You could alternatively try using CatalystX::Script::Server::Starman -  
> this should make your normal myapp_server be starman, and then try the  
> deamonize and pidfile options from the Catalyst script? Does that work  

I will try that, because it would be easier, but for the moment I have a problem with starman which seems to be leaking memory and I don't know how I can solve it.

I use Ubuntu 11 and Perl 5.14.2 and latest Catalyst and Starman and I am testing a simple MyApp Catalyst app created with:

catalyst.pl MyApp

I have compared Catalyst test server with starman and I am just following the used memory as it is reported by the top command and here are the results of my tests:

Before running the Catalyst test server:
340508k used memory

After running the Cat test server:
360992k

After the first request:
360992k

After 1000 more requests (with ab -n 1000 -c 1):
360992k

So Catalyst seems to be working fine, with no leaks.

After 1000 more requests (with ab -n 1000 -c 20):
361116k

After ~ 1 minute with no activity:
360868k

After killing Catalyst test server:
340532k

after ~ 2 minutes of inactivity:
340516k

...which is almost as before running the Catalyst test server.

After running again Catalyst test server:
360984k

After ~ 10000 requests (with ab -n 100000 -c 50):
361248k

After ~ 90000 requests:
361248k

After closing Catalyst test server:
341052k

After ~ 1 minute of inactivity:
340060k

So I can say that Catalyst and MyApp app don't leak.


After reboot, before running starman:
110192k

After running starman in a memory-friendly style, by preloading some modules, using:

starman \
--listen :5000 \
--workers 1 \
--max-requests 1000 \
--pid /srv/log/myapp.pid \
--access-log /srv/log/myapp_access.log \
--error-log /srv/log/myapp_error.log \
--daemonize \
-I /home/teddy/perl5/lib1/lib/perl5 \
-I /home/teddy/perl5/perlbrew/perls/perl-5.14.2/lib/5.14.2 \
-MMoose \
-Mnamespace::autoclean \
-MCatalyst \
-MCatalyst::Plugin::ConfigLoader \
-MCatalyst::Plugin::Static::Simple \
/srv/MyApp/myapp.psgi

144800k

After the first request:
145296k

After the second request:
145296k

After 1000 more requests (with ab -n 1000 -c 1):
146412k

After ~ 5 minutes of inactivity:
146412k

After another 1000 requests (with ab -n 1000 -c 20):
146784k

After ~ 5 minutes of inactivity:
146412k

After 10000 requests (with ab -n 100000 -c 50):
156188k

After 90000 requests:
221836k

After 100000 requests:
228572k

After ~ 15 minutes of inactivity:
227696k

After 15 more minutes of inactivity:
227696k

After one more hour of inactivity:
227944k

So it seems that there is a memory leak if the memory is not freed even after 1 hour.

I made a second test with Starman:

Before starman starts:
109944k

After starman started:
145252k

After the first request:
145392k

After 1000 requests (with ab -n 1000 -c 1):
146012k

After 1000 more requests:
146400k

After 100000 requests (with ab -n 100000 -c 1):
229864k

      After 100000 more requests (with ab -n 100000 -c 50):
320128k

After 1000000 more requests (with ab -n 1000000 -c 100):
1129368k
  
After ~ 1 hour of inactivity:
1123904k

After 1000000 more requests (with ab -n 1000000 -c 100):
1933104k

After ~ 1 hour of inactivity:
1933096k

So something's obviously wrong if so much memory is occupied even after 1 hour of inactivity.

After closing starman with kill -quit `cat myapp.pid`
1905312k

After ~ 1 hour of inactivity:
1905560k

This is even stranger, because the memory is not free even though Starman was closed, and the single solution is to reboot.

And before closing Starman I have also tried to use kill -HUP `cat myapp.pid` to reload the workers but the memory didn't decrease.

I have also tried to run starman with 5 workers and and also tried without putting it to run in the background, but it still leaks.

Does this happends only to me and others can run starman without leaks?
Or at least is there a way of limiting the memory size and make Starman restart the workers that pass that limit?

I have also tried to use:

builder {
    enable "Plack::Middleware::SizeLimit" => (
 max_unshared_size_in_kb => 1024 * 4,
 check_every_n_requests => 2
    );
    $app;
}; 

But it doesn't seem to have any effect.
I have read that "This middleware only works when the environment psgix.harakiri is set to a true value by the Plack server. If it's set to false, then this middleware simply does nothing." but I don't know if Starman sets it and supports it.

Thanks

Octavian