[Catalyst] Scalable Catalyst

[Alejandro Imass]

On Tue, Jun 30, 2009 at 2:42 PM, Tomas Doran<bobtfish at bobtfish.net> wrote:
>
> You're doing it wrong.
>
> Don't block app server threads on a remote service if you have a slow remote
> service, the only thing that lies down that route is doom and fail.
>

I don't see the problem. In fact, this was the _main and central_
point of our application: to handle as many concurrent requests acting
like a dam to the slower service. At first, it seems like a perfect
application for EDA, but any attempt in this sense failed miserably...

> Use a job queue or something so that your application servers don't sit
> waiting for slow remote services.
>

I tried POE, AnyEvent and several other EDA patterns, tests, etc.
making an external "Job Manager", and our customer even resorted to
hire someone to hack the Catalyst Engine making it even-based and it
got really ugly in the actual tests. Threads with mod_worker worked
fine and proved to be far more stable/scalable than any other option
we tried. Incredible as it sounds, multi-threaded was way more stable
than the other paths we tried.  The way I see it LWP was exactly the
solution we needed for this.

> If you really actually need to block architecturally, then a heavy weight
> application server is just the wrong solution, full stop.
>

In this case, practical proved better than theory. We heavily tested
in FreeBSD 6 and 7 and LInux 2.6.20+. The only thing we found was that
Linux's VM still needs a lot of work once you start heavily using
swap, it usually doesn't recover; it's usually faster switching than
FreeBSD 6, and even 7 but a lot less stable. FreeBSD recovered in
every case.

> I'm sure that the worker mpm will give you more headroom if you have loads
> of mod_perl processes blocking than prefork, but I very much consider this
> to be an optimisation, not a solution.
>

Ok. What would you have done? - not meant as a defensive question but
really, we would like to hear options for this application! believe
me, we tried almost everything. The event-based Catalyst engine was a
great idea, but the actual tests proved that it really had no real
advantages over process/thread with mod_worker. Besides, it proved
very problematic, unstable and hard to maintain in the long run.

I remembered my previous experiences while working on Enterprise
Service Bus technology and our results were very similar to Welsh's
final conclusions on SEDA:

"It is also worth noting that a number of recent research papers have
demonstrated that the SEDA prototype (in Java) performs poorly
compared to threaded or event-based systems implemented in C. This
would seem to contradict the findings in my work. (For more
information I invite you to read recent papers by Vivek Pai's group at
Princeton and the Capriccio work from UC Berkeley...."

Actually, no further work after SEDA has ever proved that Event-Based
is actually better than process/thread. I confirmed this with people
at Priceton (Vivek Pai himself) about a year ago.

As stated, we tried AnyEvent and POE and fiddled with the idea of
implementing some kind of Async mechanism to/from the client. It's a
B2B server so the syncing problems could have gotten very messy. We
needed to block anywhere from 1 to 7 seconds and we expected several
thousand hits per app server to make it cost effective. This is what
we came up with Catalyst:

50 Apache processes with 64 threads each for a total of 3200 threads
using about 10GB of RAM (12GB is actually on each server, for the OS,
the mutex and to avoid swapping at all costs). The 3200 blocking load
can be handled with 2 dual core AMD 64 Bit processors (4 are actually
used). not a bad deal cost-wise and it works. Each 64 thread process
grows to about 200MB RSS (> 750MB VSZ), plus a base Apache root
process of about 115MB. So we have 3200 actual blocking requests + the
Apache mutex, so the overall capacity of each app server is quite
decent for the price. At the moment we reset each process every 10000
requests with no observable or serious leaks or otherwise any sort of
problems. We take about 65ms per request in our whole processing but
sadly we have to block for the amount stated above.

The app does have some complexity because it does considerable XML
processing with LibXML 2 (DTD validations, XSL transformatiosn, XPath
queries and such), database interaction with MySQL and talks to
another server via LWP::UserAgent (this is where each thread blocks).
We chose Catalyst precisely because of the flexibility to adapt to
different scenarios (mod_perl, FastCGI, etc.) with little or no code
change.


> That said, I'm not trying to be disparaging, and I'm happy this works for
> you, and is a viable option :)
>

It did and it works fine. This is why I wouldn't discourage people to
use mod_worker, although as we have both agreed it all depends on the
actual application.

> The one thing that worries me about this is that it uses threads, and
> threads and perl don't get on.. For example, Moose's string style type

I've done many things with Perl and threads and they all have worked
fine, albeit not trivial, as long as you get around the many gotchas.
Even multi-threaded GTK2 stuff works  with no serious limitations
(long-lived threads though). In the web environment, with
multi-threaded mod_perl you have the max requests per child param so
you can work around and live with leaky/problematic modules or code.

> constraints are a bad world (because the regex used makes various versions
> of at least perl 5.8 core dump). I don't think that this is an issue for
> anything in the Catalyst stack, as we're either type-constraint free, or use
> exclusively MooseX::Types stuff (which doesn't use those regular
> expressions, and therefore is safe, I think) - but may be a problem for
> other code..
>

Yeah. This app was built before the Moose port, and I honestly don't
know much about Moose other than the fact that it was a pain in the
butt to upgrade my apps ;-) I still have to get up to speed with Moose
and the Catalyst port. We did most of the same tests to our apps after
the port and we haven't seen any major differences so we just upgraded
without much thought.

> I can certainly remember an un-fun world of apache puking it's guts on a
> coworker's machine as he was using the worker mpm..
>

A lot of people stay away from threads like the plague, but it proved
to be a better solution to this particular application. From that
experience, we have used mod_worker in other apps with similar
results, so I don't agree that it's a bad choice by any means.

> So YMMV, depending on what portion of CPAN you use, and/or what your
> codebase looks like... :-/
>

Of course. But many things are in fact thread safe or can be used in a
thread safe manner.

> Indeed - it all very much depends on the application & load profile etc - so
> all of this is just painting the bike shed, unless we're discussing a
> specific application on a real (i.e. something like production) set of
> hardware, and have actual performance metrics we want to ensure it fulfill.
> :)
>

Ok. I gave you some metrics. From a TCO standpoint, don't you think
that Catalyst, mod_perl, mpm_worker was a good choice? Homegrown
solution may have been a better solution but this was a good
compromise for us for the blocking problem.

>> BTW, Ashley suggested I write a how-to on the WIki or something like
>> that. Could some suggest exactly where, and I may have time to that
>> this week.
>
> http://dev.catalyst.perl.org/wiki/deployment
>
> Making a section linked from 'Apache' in there, briefly outlining your
> config and what benefits you get from doing mod_perl this way would be great
> :)
>

Ok. Will do. Thanks for you ample insight into this matter. For about
a year I had yet to find someone I could discuss this with!

Best,
Alejandro

BTW are you (or any others here) by any chance going to OSCON 09? If
any are, we could maybe meet there and have a Catalyst get-together
there!

> Cheers
> t0m
>
>
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>