[Catalyst] Anyone logging with RabbitMQ?

[Tomas Doran]

On 25 Feb 2012, at 00:12, Devin Austin wrote:

> Logging with RabbitMQ?  Seems like a weird thing to do but I could be
> wrong.  I use Munin to keep track of stats.

Logging != stats, logging is a superset

And you have N application servers, and you want to aggregate logs for them…

Pushing your logs into a message queue gains you a crap tonne of stuff - you have automatic aggregation (as they're all in one place), and you can have N consumers - one writing to a file, one pushing messages into elastic search, one doing bulk-inserts into a DB for usage logging, you can trivially tail the logs for N servers in one command etc..

See also:

http://logstash.net/

which should give you a nice idea of what you could do with this…

Notably, I also emit a load of more structured logging from my applications - i.e. something like:

$self->log('info', WebHit->new(
    uri => $c->req->uri,
    time_end => time(),
    time_taken => $req_time,
    memcache_hist => $req_memcache_hits,
    memcache_misses => $req_memcache_misses,
    … etc …
);

When dumped to the console, these just stringify nicely (Log::Message::Structured::Stringify::Sprintf)

The subset of messages which are specifically structured, as per above example (rather than just lines) then get pulled off RabbitMQ and stats come from them - so the same solution is nicely used for the stats collection and aggregation as well as general logging - by consuming a subset of the message queue (topic exchanges tfw!) I can draw graphs of memcache hit rate % aggregated by vhost and/or generally across all apps..

Cheers
t0m