[Catalyst-dev] RFC: JobQueue Job Notification

Wed Sep 20 10:18:13 CEST 2006

Hi Kiki

I think there are two kinds of reports back.
1) Interim progress for a long running batch. (Not sure how best to achieve
that? Previously I've logged to a run file and offered a "tail -f" web
screen view until the user navigates away, after which they can view again
from a batch view web screen or command shell)
2) Batch job completion report.

For the batch job completion report the sort of things are
- overall status (success/success-with-warning/fail)
- error details
- warning details
- log details
- full details
In addition your engine might get a 402 (etc.) code back if the handler
fails and would need to generate its own completion report in that case.

The status could go in a header X-Batch-Complete-Status or in the body.
The other items I think would need to go in the body and it might be easiest
to define a DTD and include them as an XML report. Your module could provide
routines to help the handler to generate that.
Then your jobqueue module can unpack and handle that in a standard way.

With that output I'd want to be able to 
- log selectively those different streams via a Log4perl configuration file
(crontab only really lets you do stdout and stderr)
- record the job running/complete and success status to either a database or
a file
- trigger actions on conditions, e.g. on status fail -> run a program to
send email/SMS to a sysadmin

Previously I've logged "normal" progress to one file with the volume of
messages filtered by a log level and all errors/warnings to another. I used
a batch job module to manage the process and record overall job
running/complete and success status values in SQL. I also held the batch
definitions in an SQL table.

There is a bit more complexity you could add to do with exclusivity and job
chaining.
Some batch jobs are exclusive and should only be run singly. While you can
implement this inside a job, it's better for them to call the
jobqueue/crontab replacement module to handle it, and then you can pick up
error-exited child processes and release locks and log errors if they fail
to do so.
Then there's the idea of conditional job chaining where you only run certain
jobs if their predecessors work and have alternate routes if they fail. Most
data centres need that kind of chaining for overnight batches to allow
recovery or to stop at critical points, for example if you're copy managing
large volumes of data between legacy and new systems and need to ensure they
remain consistent and in step. That's the point where crontab isn't enough
and people need to use a batch job queue package.

Regards, Peter
Dragonstaff Limited  http://www.dragonstaff.com  Business IT Consultancy

-----Original Message-----
From: catalyst-dev-bounces at lists.rawmode.org
[mailto:catalyst-dev-bounces at lists.rawmode.org] On Behalf Of Kiki
Sent: 19 September 2006 22:54
To: The elegant MVC web framework; Development of the elegant MVC web
framework
Subject: [Catalyst-dev] RFC: JobQueue Job Notification

Hello,

Anyone who's interested in the Catalyst::Engine::JobQueue, please
provide some feedback.

Here's the deal: you have a jobqueue running, jobs have been setup...
each job is sent as a $request to your app (auto, begin, controller
action, etc.), which deals with it and generates a response. This
response is returned to the Engine.

What should the engine do with it? It can email it and/or log it. Any
other ideas? How much of the response should get emailed/logged? The
full thing:
200 OK
X-Catalyst: 5.7002

<html>
...
<html>

Or just the body, body+headers,body+status?

Should the log level be computed from the response status (i.e. 100-399
- info level, 400+ - error level)?

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev at lists.rawmode.org
http://lists.rawmode.org/mailman/listinfo/catalyst-dev