[Catalyst] DProf
Octavian Rasnita
orasnita at gmail.com
Sun Nov 4 10:23:23 GMT 2007
Hi,
From: "Jon Schutz" <jon+catalyst at youramigo.com>
> All those FIN_WAIT[12] states mean that the connection is closed, and
> the process is waiting for a shutdown from the remote end. If you
> repeat the netstat command over a period of time and get the same
> processes/ports showing up, that's a problem - clients are not shutting
> down. But if you see the ports/processes continually changing, that's
> just normal.
I've seen that now (in weekend) most connections are from 66.249.72.75 (from
Google).
There are between 14 and 27 connections from that address.
My server uses KeepAlive Off, however some of those connections (with the
same IP and port) are still active even after more than half of an hour.
But some of them disappear after some more time, so I think they should not
be a problem.
> You probably want to pick a httpd that is using lots of CPU or hanging
> around longer than it should, and use lsof -p [PID], and see if that
> helps you understand what the process is doing.
I printed the lsof list and I tried to sort it by the size column, and many
tens of first items were something related to /usr/lib/locale/locale-archive
but I don't know what it means nor if it is important.
[root at octavian ~]# lsof | grep http | sort -nrk 7 | head -n 30
httpd 8020 apache mem REG 104,5 56418000
11321799 /usr/lib/locale/locale-archive
...
I also done an lsof -p [pid], and the result is below the message, but I
don't understand if it could be helpful.
Most of the httpd processes use from 12% to 15% of the processor, so I've
chosen one of them. (There are also some httpd processes that take almost
0%).
>> > Try attaching an strace to one of the busy httpds to see what system
>> > calls it is doing.
>> >
> Likewise strace -p PID on a hung/high CPU process.
I've done that, but I've just seen a continuous text like:
mremap(0x2aaac36d0000, 5115904, 5115904, MREMAP_MAYMOVE) = 0x2aaac36d0000
mremap(0x2aaac36d0000, 5115904, 5115904, MREMAP_MAYMOVE) = 0x2aaac36d0000
mremap(0x2aaac36d0000, 5115904, 5115904, MREMAP_MAYMOVE) = 0x2aaac36d0000
mremap(0x2aaac36d0000, 5115904, 5115904, MREMAP_MAYMOVE) = 0x2aaac36d0000
Process 7380 detached
> If none of the above proves useful, I'd re-suggest that you look at
> profiling the code using a mechanism other than DProf to localise which
> bit of the code is taking time.
I have also tried profiling with Apache::SmallProf but I've just seen some
lines from some perl modules printed, however most of them were simple lines
that use to assign a value to a variable, or strange things like "0;" like
this was the only content of that code line, and I couldn't see which of the
modules could have problems.
Please tell me what other options for profiling are possible.
Here is the result of lsof -p [pid]
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 7094 apache cwd DIR 104,5 4096 2 /
httpd 7094 apache rtd DIR 104,5 4096 2 /
httpd 7094 apache txt REG 104,5 328328 11311771
/usr/sbin/httpd
httpd 7094 apache mem REG 104,5 130296 11206658
/lib64/ld-2.5.so
httpd 7094 apache mem REG 104,5 615136 11206693
/lib64/libm-2.5.so
httpd 7094 apache mem REG 104,5 117656 11207001
/lib64/libpcre.so.0.0.1
httpd 7094 apache mem REG 104,5 95480 11206988
/lib64/libselinux.so.1
httpd 7094 apache mem REG 104,5 121320 11314258
/usr/lib64/libaprutil-1.so.0.2.7
httpd 7094 apache mem REG 104,5 30760 11206692
/lib64/libcrypt-2.5.so
httpd 7094 apache mem REG 104,5 241256 11320526
/usr/lib64/libldap-2.3.so.0.2.18
httpd 7094 apache mem REG 104,5 59040 11312158
/usr/lib64/liblber-2.3.so.0.2.18
httpd 7094 apache mem REG 104,5 1006192 11206823
/lib64/libdb-4.3.so
httpd 7094 apache mem REG 104,5 143272 11206986
/lib64/libexpat.so.0.5.0
httpd 7094 apache mem REG 104,5 161120 11315659
/usr/lib64/libapr-1.so.0.2.7
httpd 7094 apache mem REG 104,5 141344 11206978
/lib64/libpthread-2.5.so
httpd 7094 apache mem REG 104,5 23360 11206700
/lib64/libdl-2.5.so
httpd 7094 apache mem REG 104,5 1687464 11206682
/lib64/libc-2.5.so
httpd 7094 apache mem REG 104,5 247528 11206987
/lib64/libsepol.so.1
httpd 7094 apache mem REG 104,5 13096 11206831
/lib64/libuuid.so.1.2
httpd 7094 apache mem REG 104,5 142720 11320513
/usr/lib64/libpq.so.4.1
httpd 7094 apache mem REG 104,5 371912 11312032
/usr/lib64/libsqlite3.so.0.8.6
httpd 7094 apache mem REG 104,5 92736 11206982
/lib64/libresolv-2.5.so
httpd 7094 apache mem REG 104,5 105496 11317756
/usr/lib64/libsasl2.so.2.0.22
httpd 7094 apache mem REG 104,5 302552 11206985
/lib64/libssl.so.0.9.8b
httpd 7094 apache mem REG 104,5 1326456 11206984
/lib64/libcrypto.so.0.9.8b
httpd 7094 apache mem REG 104,5 559640 11312026
/usr/lib64/libkrb5.so.3.2
httpd 7094 apache mem REG 104,5 114352 11206840
/lib64/libnsl-2.5.so
httpd 7094 apache mem REG 104,5 174072 11312027
/usr/lib64/libgssapi_krb5.so.2.2
httpd 7094 apache mem REG 104,5 10264 11206983
/lib64/libcom_err.so.2.1
httpd 7094 apache mem REG 104,5 153008 11312025
/usr/lib64/libk5crypto.so.3.0
httpd 7094 apache mem REG 104,5 85928 11309199
/usr/lib64/libz.so.1.2.3
httpd 7094 apache mem REG 104,5 32200 11310980
/usr/lib64/libkrb5support.so.0.1
httpd 7094 apache mem REG 104,5 10400 11863608
/usr/lib64/httpd/modules/mod_auth_basic.so
httpd 7094 apache mem REG 104,5 27072 11863609
/usr/lib64/httpd/modules/mod_auth_digest.so
httpd 7094 apache mem REG 104,5 10368 11863615
/usr/lib64/httpd/modules/mod_authn_file.so
httpd 7094 apache mem REG 104,5 10400 11863610
/usr/lib64/httpd/modules/mod_authn_alias.so
httpd 7094 apache mem REG 104,5 10304 11863611
/usr/lib64/httpd/modules/mod_authn_anon.so
httpd 7094 apache mem REG 104,5 10368 11863613
/usr/lib64/httpd/modules/mod_authn_dbm.so
httpd 7094 apache mem REG 104,5 6208 11863614
/usr/lib64/httpd/modules/mod_authn_default.so
httpd 7094 apache mem REG 104,5 10400 11863620
/usr/lib64/httpd/modules/mod_authz_host.so
httpd 7094 apache mem REG 104,5 6240 11863622
/usr/lib64/httpd/modules/mod_authz_user.so
httpd 7094 apache mem REG 104,5 10368 11863621
/usr/lib64/httpd/modules/mod_authz_owner.so
httpd 7094 apache mem REG 104,5 10472 11863619
/usr/lib64/httpd/modules/mod_authz_groupfile.so
httpd 7094 apache mem REG 104,5 10400 11863617
/usr/lib64/httpd/modules/mod_authz_dbm.so
httpd 7094 apache mem REG 104,5 6208 11863618
/usr/lib64/httpd/modules/mod_authz_default.so
httpd 7094 apache mem REG 104,5 47864 11863645
/usr/lib64/httpd/modules/mod_ldap.so
httpd 7094 apache mem REG 104,5 27008 11863616
/usr/lib64/httpd/modules/mod_authnz_ldap.so
httpd 7094 apache mem REG 104,5 43552 11863643
/usr/lib64/httpd/modules/mod_include.so
httpd 7094 apache mem REG 104,5 23056 11863646
/usr/lib64/httpd/modules/mod_log_config.so
httpd 7094 apache mem REG 104,5 10384 11863648
/usr/lib64/httpd/modules/mod_logio.so
httpd 7094 apache mem REG 104,5 10360 11863635
/usr/lib64/httpd/modules/mod_env.so
httpd 7094 apache mem REG 104,5 18976 11863637
/usr/lib64/httpd/modules/mod_ext_filter.so
httpd 7094 apache mem REG 104,5 23560 11863651
/usr/lib64/httpd/modules/mod_mime_magic.so
httpd 7094 apache mem REG 104,5 10464 11863636
/usr/lib64/httpd/modules/mod_expires.so
httpd 7094 apache mem REG 104,5 18784 11863631
/usr/lib64/httpd/modules/mod_deflate.so
httpd 7094 apache mem REG 104,5 18784 11863640
/usr/lib64/httpd/modules/mod_headers.so
httpd 7094 apache mem REG 104,5 14560 11863666
/usr/lib64/httpd/modules/mod_usertrack.so
httpd 7094 apache mem REG 104,5 14528 11863660
/usr/lib64/httpd/modules/mod_setenvif.so
httpd 7094 apache mem REG 104,5 18712 11863650
/usr/lib64/httpd/modules/mod_mime.so
httpd 7094 apache mem REG 104,5 89176 11863628
/usr/lib64/httpd/modules/mod_dav.so
httpd 7094 apache mem REG 104,5 18680 11863662
/usr/lib64/httpd/modules/mod_status.so
httpd 7094 apache mem REG 104,5 35424 11863623
/usr/lib64/httpd/modules/mod_autoindex.so
httpd 7094 apache mem REG 104,5 18976 11863644
/usr/lib64/httpd/modules/mod_info.so
httpd 7094 apache mem REG 104,5 47864 11863629
/usr/lib64/httpd/modules/mod_dav_fs.so
httpd 7094 apache mem REG 104,5 10368 11863668
/usr/lib64/httpd/modules/mod_vhost_alias.so
httpd 7094 apache mem REG 104,5 31296 11863652
/usr/lib64/httpd/modules/mod_negotiation.so
httpd 7094 apache mem REG 104,5 10360 11863632
/usr/lib64/httpd/modules/mod_dir.so
httpd 7094 apache mem REG 104,5 10368 11863605
/usr/lib64/httpd/modules/mod_actions.so
httpd 7094 apache mem REG 104,5 14592 11863661
/usr/lib64/httpd/modules/mod_speling.so
httpd 7094 apache mem REG 104,5 10400 11863665
/usr/lib64/httpd/modules/mod_userdir.so
httpd 7094 apache mem REG 104,5 14552 11863606
/usr/lib64/httpd/modules/mod_alias.so
httpd 7094 apache mem REG 104,5 60416 11863659
/usr/lib64/httpd/modules/mod_rewrite.so
httpd 7094 apache mem REG 104,5 72856 11863653
/usr/lib64/httpd/modules/mod_proxy.so
httpd 7094 apache mem REG 104,5 22944 11863655
/usr/lib64/httpd/modules/mod_proxy_balancer.so
httpd 7094 apache mem REG 104,5 31424 11863657
/usr/lib64/httpd/modules/mod_proxy_ftp.so
httpd 7094 apache mem REG 104,5 27232 11863658
/usr/lib64/httpd/modules/mod_proxy_http.so
httpd 7094 apache mem REG 104,5 10400 11863656
/usr/lib64/httpd/modules/mod_proxy_connect.so
httpd 7094 apache mem REG 104,5 31320 11863624
/usr/lib64/httpd/modules/mod_cache.so
httpd 7094 apache mem REG 104,5 10328 11863663
/usr/lib64/httpd/modules/mod_suexec.so
httpd 7094 apache mem REG 104,5 22912 11863633
/usr/lib64/httpd/modules/mod_disk_cache.so
httpd 7094 apache mem REG 104,5 10672 11863638
/usr/lib64/httpd/modules/mod_file_cache.so
httpd 7094 apache mem REG 104,5 27168 11863649
/usr/lib64/httpd/modules/mod_mem_cache.so
httpd 7094 apache mem REG 104,5 27256 11863626
/usr/lib64/httpd/modules/mod_cgi.so
httpd 7094 apache mem REG 104,5 17944 11864601
/usr/lib64/httpd/modules/mod_apreq2.so
httpd 7094 apache mem REG 104,5 53144 11311822
/usr/lib64/libapreq2.so.3.6.1
httpd 7094 apache mem REG 104,5 1383702 11864603
/usr/lib64/httpd/modules/mod_perl.so
httpd 7094 apache mem REG 104,5 1260624 11406582
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
httpd 7094 apache mem REG 104,5 18152 11206997
/lib64/libutil-2.5.so
httpd 7094 apache mem REG 104,5 35424 11863654
/usr/lib64/httpd/modules/mod_proxy_ajp.so
httpd 7094 apache mem REG 104,5 169288 11863687
/usr/lib64/httpd/modules/mod_ssl.so
httpd 7094 apache mem REG 104,5 18632 11310326
/usr/lib64/libdistcache.so.1.0.1
httpd 7094 apache mem REG 104,5 39040 11312048
/usr/lib64/libnal.so.1.0.1
httpd 7094 apache mem REG 104,5 129968 11864600
/usr/lib64/httpd/modules/mod_dav_svn.so
httpd 7094 apache mem REG 104,5 131896 2261104
/usr/lib64/libsvn_repos-1.so.0.0.0
httpd 7094 apache mem REG 104,5 22144 2261090
/usr/lib64/libsvn_fs-1.so.0.0.0
httpd 7094 apache mem REG 104,5 41408 2261086
/usr/lib64/libsvn_delta-1.so.0.0.0
httpd 7094 apache mem REG 104,5 214336 2261106
/usr/lib64/libsvn_subr-1.so.0.0.0
httpd 7094 apache mem REG 104,5 115744 2261094
/usr/lib64/libsvn_fs_fs-1.so.0.0.0
httpd 7094 apache mem REG 104,5 173856 2261092
/usr/lib64/libsvn_fs_base-1.so.0.0.0
httpd 7094 apache mem REG 104,5 11264 11864599
/usr/lib64/httpd/modules/mod_authz_svn.so
httpd 7094 apache mem REG 104,5 53880 11206683
/lib64/libnss_files-2.5.so
httpd 7094 apache mem REG 104,5 56418000 11321799
/usr/lib/locale/locale-archive
httpd 7094 apache mem REG 104,5 78304 12127032
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/APR/Pool/Pool.so
httpd 7094 apache mem REG 104,5 148688 12156930
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/RequestRec/RequestRec.so
httpd 7094 apache mem REG 104,5 45861 11537397
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache/DB/DB.so
httpd 7094 apache mem REG 104,5 19248 11436188
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
httpd 7094 apache mem REG 104,5 13408 11436173
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
httpd 7094 apache mem REG 104,5 21776 11436363
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
httpd 7094 apache mem REG 104,5 50760 11436570
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
httpd 7094 apache mem REG 104,5 97247 11436574
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
httpd 7094 apache mem REG 104,5 31736 11436136
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
httpd 7094 apache mem REG 104,5 21328 11436396
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
httpd 7094 apache mem REG 104,5 19800 11436176
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
httpd 7094 apache mem REG 104,5 390058 11665829
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
httpd 7094 apache mem REG 104,5 12912 11436197
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
httpd 7094 apache mem REG 104,5 38592 11436161
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so
httpd 7094 apache mem REG 104,5 111633 11831033
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Set/Object/Object.so
httpd 7094 apache mem REG 104,5 244746 6750210
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
httpd 7094 apache mem REG 104,5 73029 11864687
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
httpd 7094 apache mem REG 104,5 153479 11665875
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
httpd 7094 apache mem REG 104,5 93027 12156970
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/Connection/Connection.so
httpd 7094 apache mem REG 104,5 40340 12156958
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/Const/Const.so
httpd 7094 apache mem REG 104,5 114832 12127058
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/RequestIO/RequestIO.so
httpd 7094 apache mem REG 104,5 116780 12156946
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/RequestUtil/RequestUtil.so
httpd 7094 apache mem REG 104,5 79545 12156949
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/Response/Response.so
httpd 7094 apache mem REG 104,5 62514 12127064
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Apache2/URI/URI.so
httpd 7094 apache mem REG 104,5 83688 12127010
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/APR/Table/Table.so
httpd 7094 apache mem REG 104,5 92840 12029188
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Template/Stash/XS/XS.so
httpd 7094 apache mem REG 104,5 71467 11931052
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/C3/XS/XS.so
httpd 7094 apache mem REG 104,5 324539 11995001
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBI/DBI.so
httpd 7094 apache mem REG 104,5 119304 11436201
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
httpd 7094 apache mem REG 104,5 15816 11436146
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
httpd 7094 apache mem REG 104,5 289130 11831125
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Date/Calc/Calc.so
httpd 7094 apache mem REG 104,5 53398 12419162
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/LibXML/Common/Common.so
httpd 7094 apache mem REG 104,5 1301232 11310209
/usr/lib64/libxml2.so.2.6.29
httpd 7094 apache mem REG 104,5 293984 12419334
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/LibXML/LibXML.so
httpd 7094 apache DEL REG 0,8 19782488
/dev/zero
httpd 7094 apache mem REG 104,5 25464 11371437
/usr/lib64/gconv/gconv-modules.cache
httpd 7094 apache DEL REG 0,8 19782489
/dev/zero
httpd 7094 apache DEL REG 0,8 19612453
/dev/zero
httpd 7094 apache mem REG 104,5 116488 11665591
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so
httpd 7094 apache mem REG 104,5 1505832 11436589
/usr/lib64/mysql/libmysqlclient.so.15.0.0
httpd 7094 apache 0u sock 0,5 19782614 can't
identify protocol
httpd 7094 apache 1u unix 0xffff81020ffbd100 19782498 socket
httpd 7094 apache 2w REG 104,5 9427311 14615226
/var/log/httpd/error_log
httpd 7094 apache 3r CHR 1,9 2239
/dev/urandom
httpd 7094 apache 5u sock 0,5 19612397 can't
identify protocol
httpd 7094 apache 7u sock 0,5 19612402 can't
identify protocol
httpd 7094 apache 8r FIFO 0,6 19782485 pipe
httpd 7094 apache 9w FIFO 0,6 19782485 pipe
httpd 7094 apache 10w REG 104,5 9427311 14615226
/var/log/httpd/error_log
httpd 7094 apache 11w REG 104,5 0 4555243
/oct/TranzactiiBursiere/logs/error_log
httpd 7094 apache 12w REG 104,5 10080 14615228
/var/log/httpd/ssl_error_log
httpd 7094 apache 13w REG 104,5 0 14614970
/var/log/httpd/access_log
httpd 7094 apache 14w REG 104,5 9427311 14615226
/var/log/httpd/error_log
httpd 7094 apache 15w REG 104,5 2616603 4557215
/oct/TranzactiiBursiere/logs/access_log
httpd 7094 apache 16w REG 104,5 0 14614971
/var/log/httpd/ssl_access_log
httpd 7094 apache 17w REG 104,5 0 14614983
/var/log/httpd/ssl_request_log
httpd 7094 apache 18r 0000 0,10 0 19782492
eventpoll
httpd 7094 apache 19w CHR 1,3 1801
/dev/null
httpd 7094 apache 20r CHR 1,3 1801
/dev/null
Thank you very much for help.
Octavian
More information about the Catalyst
mailing list