[Catalyst] Job Queue recommendation

Bill Moseley moseley at hank.org
Sat Jan 22 16:01:19 GMT 2011


On Fri, Jan 21, 2011 at 11:41 PM, Cosimo Streppone <cosimo at streppone.it>wro=
te:

> There seems to be two common approaches. One is to just start up a bunch =
of
>> worker processes independently (managed like any other daemon) killing a=
nd
>> starting more as needed.
>>
>
> For some time I used this approach, with a "fake" init
> script like '/etc/init.d/worker' that would start one of them,
> so I could run it multiple times to spawn more workers.
>
> A munin plugin monitored the number of active workers,
> and if it detected less than normal, it would spawn a new
> one.
>
> I thought that'd be neat, then I changed my mind.
> It worked just fine, but in the end I didn't like
> mixing munin plugins with application-related stuff.
>

What did you change to using?

IMO, we (at work) have tended to over-engineer the workers.  For example,
for a given task we have servers that run via init.d scripts and they fork
off a number of threads to handle connections and also fork off a bunch of
worker threads that another manager thread allocates work to.

It's a complex beast just to run some code.  And as a beast if it breaks
there's only one or two people that can fix.  When the on-call tech gets a
call at 2am it's unlikely they would know how to restart specifying more
threads.

The advantage I see in using separate scripts, like you describe above, is
simplicity.  If a human was alerted that a queue was getting large then it's
just a matter of finding a machine that has headroom and start a few more
workers.  Any operator can do that.  Plus, that's something that could be
automated pretty easily with a shell script (and maybe even run via cron).

A disadvantage is Perl is not the fastest in startup time (compared to
forking, for example), but hopefully starting extra processes would be a
rare situation when load changed dramatically.


 BTW -- Celery is worth a look:  http://celeryq.org/





-- =

Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20110122/c0949=
69d/attachment.htm


More information about the Catalyst mailing list