[Catalyst-dev] RFC: JobQueue Job Notification

[Matt S Trout]

Kiki wrote:
> Peter Edwards wrote:
>> There is a bit more complexity you could add to do with exclusivity and job
>> chaining.
>> Some batch jobs are exclusive and should only be run singly. While you can
>> implement this inside a job, it's better for them to call the
>> jobqueue/crontab replacement module to handle it, and then you can pick up
>> error-exited child processes and release locks and log errors if they fail
>> to do so.
>> Then there's the idea of conditional job chaining where you only run certain
>> jobs if their predecessors work and have alternate routes if they fail. Most
>> data centres need that kind of chaining for overnight batches to allow
>> recovery or to stop at critical points, for example if you're copy managing
>> large volumes of data between legacy and new systems and need to ensure they
>> remain consistent and in step. That's the point where crontab isn't enough
>> and people need to use a batch job queue package.
>>
>>   
> Hmm, while I see that this is quite the standard for batch job systems,
> I don't think this Engine quite fits the description. The jobs for this
> engine are supposed to do lightweight stuff, be modeled on HTTP
> request-response and to be related to the Web Application not system
> maintenance.
> 
> I'm not excluding the possibility to implement semaphores/locks and
> chaining, but this would be very low priority...

Locks are potentially important - or being able to say "only one run of job 
type X at once" at least.

Job chaining shouldn't be the responsibility of the job runner - since we're 
already intending a $c->schedule interface for apps to request things to be 
run, a workflow job could simply schedule its parts via a wrapper that invokes 
the inner job directly ($c->subreq would perhaps pretty much do the trick if 
we can get trunk SubRequest to CPAN) and then schedules an appropriate next 
stage itself.

-- 
      Matt S Trout       Offering custom development, consultancy and support
   Technical Director    contracts for Catalyst, DBIx::Class and BAST. Contact
Shadowcat Systems Ltd.  mst (at) shadowcatsystems.co.uk for more information

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +