[Catalyst-dev] Catalyst-Engine-JobQueue-POE

Kiki kiki at bsdro.org
Fri Sep 15 20:55:08 CEST 2006


Jonathan Rockway wrote:
>> This engine is meant to be used by a separate process than the webserver
>> (an example can be found in t/testapp_jobqueue.pl) to schedule actions
>> to be executed at certain points in time (a al cron).
>>
>> This is more a proof-of-concept implementation to see how things would
>> work out with POE and mst's idea of the execution mechanism. T
>>     
>
> This is what we were talking about in #catalyst a few days ago, right?
>
> TODO:
>
> - get crontab info from myapp_config, if it exists
>   
Doable, though i really prefer the crontab syntax for specifying the
schedule times, provides a nicer overview of what runs when, but otoh
the job data (URI + arguments) look kinda hackish in there... perhaps a
combo could be be useful, well, TBPU (To Be Pondered Upon)

I think that it should look either for your schedule info (as per your
example below), getting it directly from myapp_config or for a
schedule_file option which would then be parsed as a crontab file.
> - IPC, probably as catalyst plugin.
>  - reason: so we can do $c->schedule('in 15 minutes', $foo);
>   
Yes, there will be some kind of IPC. IMO, this needs an comm protocol
(and a basic API) specified first, and we need to decide on how much of
the request info must be supplied for each job to nail down the
protocol. Then we can start throwing shared memory, socket and db-mapped
plugins at it.
>  - auto-start the server when myapp_server/fastcgi/mod_perl runs
>   
Well, I dunno how well POE plays with Proc::Daemon et al. I'll look into
it. Afaik there's not much spawning of processes going on in Catalyst so
I take it there's no recommended practice. More than that, I'm kinda
uneasy with apache and/or fastcgi processes spawning the jobqueue runner
(or any kind of long running daemon), cause it's kinda unexpected for
them to do so.
>> - there is no documentation (though the Engine is heavily inspired by
>> the HTTP::POE engine)
>> - the schedule is parsed from a crontab-like file (an example can be
>> found in t/crontab
>> - there is no support for authz (but planned for a future version)
>> - the scheduling resolution is 1 minute
>> - no support for at-like jobs (one-time execution) (but planned for a
>> future version)
>> - uncaught exceptions cause the process to die silently (ex: not finding
>> a crontab file to parse)
>>     
>
> Silently?  We should probably use $c->log.
>
>   
Yes, this looks like a bug to me, the problem is that  the job queue
lacks the IO framework that a webserver provides, so I kinda left this
as a TODO. I'm sure it can be resolved. Dunno why it craps out without a
sound, it should be croaking left and right...
>> Crontab syntax
>>
>> Fields are whitespace separated
>> The first 5 fields are a standard cronspec (man 5 crontab)
>> The next field is a username (will be used for authz)
>> The next field is a URI (used for dispatching)
>> The rest of the fields are arguments (they are concatenated with '&' and
>> passed as QUERY_STRING)
>>     
>
> Why query string?  Why not just properly $c->execute with the various
> arguments?  Nobody wants to urlencode their config file.  I will
> probably make the config something like this:
>
> Schedule:
>   - 3:30 everday
>     - execute: /foo/bar
>         arguments:
>           - baz
>           - quux
>     - execute: /foo/cleanup
>         arguments:
>           - hello
>           - world
>
> or something similar
>
>   
Well, you surely would want to have values for those arguments too:
          
            - execute: /foo/bar
                    arguments
                        baz: some_value
                        quux: some_other_value

The point here is not the url encoding, but the syntax we should use to
give the job the data for the action. Right now this is based on
Catalyst::Engine::CGI, but if it is warranted, I might build the whole
shebang (the request) by myself (but first, I need to know where to get
the info from and how to do it). Your example is as good as mine (only
it would read:
 30 3 * * * www /foo/bar baz quux
 30 3 * * * www /foo/cleanup hello world
)

Is this really sufficient? Or do we need more info.
>> As with C::E::HTTP::POE sending an USR1 signal to the jobqueue process
>> will cause it to dump state.
>>     
>
> Can you elaborate on this?
>
>   
It's a debug thing, it dumps the object and some info to stderr (if you
want to look into its guts: kill -USR1 <job_queue_runner_pid>)




More information about the Catalyst-dev mailing list