[Catalyst] DProf

[Jon Schutz]

On Sat, 2007-11-03 at 16:01 +0200, Octavian Rasnita wrote:
> From: "Jon Schutz" <jon+catalyst at youramigo.com>
> 
> > If benchmark testing isn't showing the fault, it's likely that it is
> > data-dependent, i.e. it depends on something in the request or response
> > - something that the benchmark doesn't trigger.
> >
> > Perhaps clients are left lingering - an incorrect Content-Length could
> > cause that.  I've also seen apache get stuck in busy wait states when
> > proxying through to a remote site that is not responding correctly.
> > Your top output is very likely an indication that some requests are not
> > being served to completion.
> 
> I've tried to put ab to run more concurent connections in order to check how 
> occupied will be the processor, and with:
> 
> ab -n 1000 -t 1000 -c 35 [URL]
> 
> the processor was occupied sometimes 4 to 9% and sometimes 49 to 80%. Now is 
> weekend, and there are very very few visitors on my site, and I can assume 
> that my requests were the only requests, so no other strange requests target 
> my server.
> 
> > You could try (as root)
> >
> > netstat -nap | grep http
> 
> I've tried and the latests results are below the message, but I can't 
> interpret them. I am not a very good Linux user unfortunately.
> 

All those FIN_WAIT[12] states mean that the connection is closed, and
the process is waiting for a shutdown from the remote end.  If you
repeat the netstat command over a period of time and get the same
processes/ports showing up, that's a problem - clients are not shutting
down.  But if you see the ports/processes continually changing, that's
just normal.


> > to see what network connections the httpd processes are holding open.
> >
> > lsof may also tell you something useful.
> 
> I have also ran that command and grepped the results that contain "http" and 
> from all the results ~ 5000, aproximately 4000 contain "http".
> 

You probably want to pick a httpd that is using lots of CPU or hanging
around longer than it should, and use lsof -p [PID], and see if that
helps you understand what the process is doing.

> > Try attaching an strace to one of the busy httpds to see what system
> > calls it is doing.
> >

Likewise strace -p PID on a hung/high CPU process.

> > The difficulty with the mod_perl environment is that your httpd
> > processes are polluted with everything else that is happening on your
> > server - PHP etc - so what perl is responsible for is obscured.  Testing
> > in a fastcgi environment is better in this respect since perl and apache
> > are separated.  If you are able to try fastcgi, it might reveal some
> > further insights.
> 
> I have never used FastCGI so for the moment I am not able to try it right 
> now.
> 

If none of the above proves useful, I'd re-suggest that you look at
profiling the code using a mechanism other than DProf to localise which
bit of the code is taking time.

-- 

Jon