[Catalyst] Anyone logging with RabbitMQ?

[Tomas Doran]

On 25 Feb 2012, at 00:04, Bill Moseley wrote:

> Anyone have experience with this?  What tool are you using to aggregate (and monitor, or generate stats with)?  Any tips?

I was doing this for a while.

However it gives you a single point of failure / slowness. The module you mention is entirely synchronous, so if your RabbitMQ server ever gets slow or sick, then your web app is going to stop working very well…

The way I do this at work is to run a tiny daemon on each web server that has a ZMQ sub socket listener. The web application processes connect to this daemon via ZMQ pub socket, and the daemon just casts any messages received up to Rabbit.

ZMQ is really really fast, and also it's pub/sub sockets have exactly the right semantics (i.e. they can be asked to do a selective amount of queuing, but to drop messages rather than block if the upstream is too slow). This works brilliantly, but there are a couple of gotchas as you have to be _very_ careful to not be using ZMQ across fork() calls - which implies a custom FCGI process manager etc..

For a simpler solution, casting each line using syslog on localhost, and then aggregating that to RabbitMQ would work quite well..

Cheers
t0m

P.S. I do plan to open source code to implement the above in the next few months, but I'm not there yet, sorry…