[Catalyst] Memory usage with FCGI? Server running out of RAM

[Lasse Makholm]

In my experience, it's usually a few naughty requests that gobble up a lot
of memory. We found it useful to record process vsize and rss at the
beginning and end of the request cycle to generate warnings like this:

Warning: PID 28710 grew rss by 20% from 42 MiB to 51 MiB serving GET
/api/search?q=foo

One of the common mistakes in our app was inflating all row objects in a
result set at the same time when it wasn't needed. We prefetch a lot so row
objects and their related objects get big. In those cases, this:

foreach my $row ($rs->all)
{
push @results, $row->to_json,
}

Will use a lot more memory than this:

while (my $row = $rs->next)
{
push @results, $row->to_json,
}

DBIx::Class's HashRefInflator is often touted for its speed but it's also a
lot lighter on memory...

Another thing you could consider is using Plack to run all your Perl apps
in the same server/process. Memory for all common dependencies would be
shared among the apps in that case which might get you a decent gain...

There's also a bunch of Plack::MiddleWare's on CPAN to help with
debugging/profiling of apps...

/L



On Thu, Jun 11, 2015 at 3:03 PM, Denny <2015 at denny.me> wrote:

> I run a server with 18 separate installs of ShinyCMS running, with at
> least 3 and sometimes 20 processes per install. 122 perl-fcgi processes in
> total. They look a bit bigger than most of your processes. I've got 8GB of
> RAM, and I'm pushing up against the edges of it a bit, but I don't usually
> have any problems.
>
> Hope that's useful info!
>
>
>
> On 11 June 2015 13:31:16 BST, Jesse Sheidlower <jester at panix.com> wrote:
>
>>
>> Sysadmin-fu needed: I have a personal server in the cloud that has 1 GB
>> of RAM. It's running Debian, and the usual server basics--MySQL, nginx,
>> etc. No X, of course.
>>
>> There are three Catalyst apps running, all of them fairly small, using
>> nginx and FCGI (I run the MyApp_fastcgi.pl script to a socket, that
>> nginx then responds to). But they're taking up all the memory on the
>> system, and I can't run another Cat app (and sometimes even system
>> commands hang for lack of memory). I tried reducing --nproc to 1 for two
>> of the three apps that I'm the only one using (the other has --nproc 3).
>> The output of "top" with "M" (sorting by RES) looks like (I removed a
>> few columns for spacing):
>>
>>   PID USER     VIRT  RES  SHR S  %MEM    TIME+  COMMAND
>>
>> 23766 www-data 440m 230m 2584 S  23.0   0:13.40 perl-fcgi
>> 16958 mysql    428m 188m    0 S  18.8 118:14.98
>> mysqld
>> 25992 www-data 375m 158m 2608 S  15.9   0:06.80 perl-fcgi
>> 26884 www-data 238m  80m 2616 S  8.0   0:01.61  perl-fcgi
>>  2513 www-data 282m  76m    0 S  7.7   0:06.01  perl-fcgi
>>  2512 www-data 182m  66m    0 S  6.6   0:00.00  perl-fcgi-pm [Library]
>>  1825 www-data 172m  61m    0 S  6.1   0:03.44  perl
>>  2487 www-data 163m  46m   60 S  4.6   0:00.18  perl-fcgi-pm [CatSF]
>>  1692 root     119m 5128  140 S  0.5   0:56.32  rsyslogd
>> 19998 www-data 79452 3296 1136 S 0.3   6:10.61  nginx
>>  2265 www-data 79768 3148  504 S 0.3  15:57.81  nginx
>> [...]
>>
>> Is this an expected or appropriate amount of memory usage? I appreciate
>> that 1GB isn't all that much memory nowadays, but this is a very lightly
>> loaded machine, and I don't want to pay a lot more for a larger cloud
>> server to handle this if it's not necessary.
>>
>> Thanks.
>>
>> ------------------------------
>>
>> List: Catalyst at lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
> --
> Sent from my phone. Please excuse typos, terseness, and top-posting.
>
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scsys.co.uk/pipermail/catalyst/attachments/20150611/e1a0acb5/attachment.htm>