[Catalyst] Re: Catalyst benchmark 5.7 VS 5.8

[Fayland Lam]

Thanks for that. (Toby Corkindale too)

I'll do more research and update you later.

Thanks.

Andrew Rodland wrote:
> On Monday 28 September 2009 12:56:36 am Fayland Lam wrote:
>> I'm wondering if someone here did a benchmark between Catalyst 5.7 and 5.8
> 
> Benchmark, as requested. View this message at http://p3m.org/pfn/3499 if your 
> mailer is too high-tech for fixed-width text.
> 
> 
> The Setup:
> 
> Linux 2.6 OpenVZ, on Quad 2.2GHz Opteron.
> Perl 5.10.0 + debian patches.
> FastCGI via Apache 2.2, mpm_event, mod_fastcgi.
> The two test instances were running on the same machine, with the same perl
> and the same checkout of the app, but different local::lib directories.
> 
> FastCGI was set to 10 processes. The page I was hitting was from a real
> checkout of a real production app, however it was the front page of the
> site, which is fairly light on dynamic content. I figured this was
> appropriate since it would better show any differences in Catalyst rather
> than spending a lot of time in the backend. The code still hits several
> models, 3 actions, and a view, but perhaps it was a little too fast since,
> as you'll see below, my throughput was ultimately limited by the number of
> running processes. Each instance was given a "warmup" run (the results of
> which were discarded) before the following tests were run. My tool collects
> statistics on the return status, but for all tests the returns were all 200
> (success) so I've left out that row. 
> 
> || 20 requests/second (20 threads) for 60s
> | Metric            || Catalyst 5.7010          || Catalyst 5.8011
> |===================||==========================||==========================
> | Hits              || 1200                     || 1200
> | Throughput        || 20.00 req/s              || 20.00 req/s
> | Latency (mean)    || 0.072s                   || 0.074s
> | Latency (SD)      || 0.013s                   || 0.017s
> | Latency (Q1-Q3)   || 0.064 - 0.078s           || 0.066 - 0.080s
> 
> || 40 requests/second (40 threads) for 60s
> | Metric            || Catalyst 5.7010          || Catalyst 5.8011
> |===================||==========================||==========================
> | Hits              || 2400                     || 2400
> | Throughput        || 40.00 req/s              || 40.00 req/s
> | Latency (mean)    || 0.083s                   || 0.088s
> | Latency (SD)      || 0.020s                   || 0.024s
> | Latency (Q1-Q3)   || 0.069 - 0.095s           || 0.072 - 0.100s
> 
> || 80 requests/second (80 threads) for 60s
> | Metric            || Catalyst 5.7010          || Catalyst 5.8011
> |===================||==========================||==========================
> | Hits              || 4675                     || 4637
> | Throughput        || 77.92 req/s              || 77.28 req/s
> | Latency (mean)    || 0.688s                   || 0.708s
> | Latency (SD)      || 0.178s                   || 0.187s
> | Latency (Q1-Q3)   || 0.617 - 0.800s           || 0.726 - 0.811s
> 
> The difference between 5.7 and 5.8 in these results is consistently in favor
> of 5.7, but by a margin of between 0% and 5% which is not a whole lot in my
> book. By my unscientific measure (i.e. looking at "top") of memory usage,
> 5.7 used 138MB of RAM (for fcgi-pm + 10x fcgi children) whereas 5.8 used
> 184MB, so that's a 33% expansion, which is a more significant issue. I have
> a feeling that most of that is shared, and so the difference wouldn't
> increase much with an increase in the number of processes, but I haven't
> investigated that yet.
> 
> Questions?
> 
> Andrew "hobbs" Rodland
> 
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>