[Catalyst] Zeus and Catalyst

Duncan Garland Duncan.Garland at motortrak.com
Wed Jan 12 09:02:20 GMT 2011


Hi T0m,

Thanks, that's a very detailed email with a lot of ideas. It'll take a while to go through all of them.

You mentioned gdb. We've got a crontab which releases the locked process and one of the first things we did was to make gdb do a dump before killing the process. However, we haven't been able to interpret the output properly. I've attached an example below. It appears to be preparing an SQL statement but it doesn't give us much idea about which one.

Have you (or anyone else) got any idea what all this means?

not ok Sat Jan  8 18:23:01 2011: http://www.xxxx.co.uk/use_ws_1
500 read timeout
11842 perl /usr/local/etc/httpd/fcgi/catalyst/xxxx/script/xxxx_fastcgi.pl
not ok Sat Jan  8 18:23:01 2011: http://www.xxxx.co.uk/use_ws_2
500 read timeout
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 11842
Reading symbols from /usr/bin/perl...(no debugging symbols found)...done.
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Util/Util.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Util/Util.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Name/Name.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Devel/GlobalDestruction/GlobalDestruction.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Devel/GlobalDestruction/GlobalDestruction.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/MOP/MOP.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/MOP/MOP.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Moose/Moose.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Moose/Moose.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Variable/Magic/Magic.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Variable/Magic/Magic.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Identify/Identify.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Identify/Identify.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/MooseX/Role/WithOverloading/WithOverloading.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/MooseX/Role/WithOverloading/WithOverloading.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cache/FastMmap/CImpl/CImpl.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cache/FastMmap/CImpl/CImpl.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/FCGI/FCGI.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/FCGI/FCGI.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/XSAccessor/XSAccessor.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/XSAccessor/XSAccessor.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Date/Calc/Calc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Date/Calc/Calc.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/encoding/encoding.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/encoding/encoding.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBI/DBI.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBI/DBI.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/Oracle/Oracle.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/Oracle/Oracle.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1...(no debugging symbols found)...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
Reading symbols from /oracle/product/10.1.0/db_1/lib/libnnz10.so...(no debugging symbols found)...done.
Loaded symbols for /oracle/product/10.1.0/db_1/lib/libnnz10.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
Reading symbols from /usr/local/lib/libexpat.so.0...done.
Loaded symbols for /usr/local/lib/libexpat.so.0
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Math/BigInt/FastCalc/FastCalc.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Math/BigInt/FastCalc/FastCalc.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Image/Magick/Magick.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Image/Magick/Magick.so
Reading symbols from /usr/lib64/libMagickCore.so.1...done.
Loaded symbols for /usr/lib64/libMagickCore.so.1
Reading symbols from /usr/lib64/libMagickWand.so.1...done.
Loaded symbols for /usr/lib64/libMagickWand.so.1
Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libtiff.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libtiff.so.3
Reading symbols from /usr/lib64/libjpeg.so.62...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libjpeg.so.62
Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfontconfig.so.1
Reading symbols from /usr/lib64/libXext.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXext.so.6
Reading symbols from /usr/lib64/libSM.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libSM.so.6
Reading symbols from /usr/lib64/libICE.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libICE.so.6
Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /usr/lib64/libXt.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXt.so.6
Reading symbols from /usr/lib64/libbz2.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libbz2.so.1
Reading symbols from /usr/local/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libz.so.1
Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/SDBM_File/SDBM_File.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/SDBM_File/SDBM_File.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/String/String.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/String/String.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map/Map.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map/Map.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map8/Map8.so...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map8/Map8.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Unicode/Unicode.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Unicode/Unicode.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/JP/JP.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/JP/JP.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/B/B.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/B/B.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Clone/Clone.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Clone/Clone.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so
Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Template/Stash/XS/XS.so...done.
Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Template/Stash/XS/XS.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so
0x0000003bbe20d8e0 in __read_nocancel () from /lib64/libpthread.so.0
(gdb) (gdb) #0  0x0000003bbe20d8e0 in __read_nocancel () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00002ac7089512a2 in snttread () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#2  0x00002ac70894e42a in ?? () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#3  0x00002ac708818d42 in nsprecv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#4  0x00002ac70881c8e3 in nsrdr () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#5  0x00002ac7087f6cc8 in nsdo () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#6  0x00002ac7087f662a in nsbrecv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#7  0x00002ac708831c0d in nioqrc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#8  0x00002ac708992985 in ttcdrv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#9  0x00002ac70883a4fc in nioqwa () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#10 0x00002ac708657c4f in upirtrc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#11 0x00002ac7085c1193 in kpurcsc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#12 0x00002ac70856fbcf in kpuexecv8 () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#13 0x00002ac70856d7b3 in kpuexec () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#14 0x00002ac70861bb27 in OCIStmtExecute () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1
No symbol table info available.
#15 0x00002ac70803adc9 in ora_describe (h=0x108af110, imp_sth=0x15fdfa10) at oci8.c:1997
        my_perl = <value optimized out>
        imp_dbh = 0x15594520
        long_readlen = 6553500
        num_fields = <value optimized out>
        num_errors = <value optimized out>
        has_longs = <value optimized out>
        est_width = <value optimized out>
        nested_cursors = <value optimized out>
        i = <value optimized out>
        status = <value optimized out>
#16 0x00002ac70803e274 in ora_st_prepare (sth=0x108af110, imp_sth=0x15fdfa10, statement=<value optimized out>, attribs=<value optimized out>) at oci8.c:337
        my_perl = <value optimized out>
        imp_dbh = <value optimized out>
        status = 0
        oparse_lng = 1
        ora_check_sql = 1
        ora_placeholders = 48
        ora_pad_empty = 0x15519200
#17 0x00002ac7080215c1 in XS_DBD__Oracle__st__prepare (my_perl=0xd8b8010, cv=<value optimized out>) at Oracle.xsi:434
        imp_sth = 0x15fdfa10
        sth = 0x108af110
        statement = 0x107204b0
        attribs = 0x0
        sp = <value optimized out>
        ax = <value optimized out>
        items = <value optimized out>
#18 0x0000003bc2e90a96 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#19 0x0000003bc2e8a33e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#20 0x0000003bc2e37576 in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#21 0x00002ac707de97da in XS_DBI_dispatch (my_perl=0xd8b8010, cv=0xef1e7e0) at DBI.xs:3459
        sp = 0x0
        ax = 6
        items = 2
        perinterp_sv = <value optimized out>
        PERINTERP = 0x10740aa0
        h = 0xff2bdc0
        st1 = 0x10742000
        st2 = 0xef1e7e0
        err_sv = <value optimized out>
        tmp_svp = <value optimized out>
        hook_svp = 0xc0d8b8010
        mg = <value optimized out>
        gimme = 2064
        trace_flags = 0
        trace_level = 0
        is_DESTROY = 0
        is_unrelated_to_Statement = 0
        keep_error = 0
        ErrCount = 18446744073709551615
        i = 227246096
        outitems = 228021872
        call_depth = 1
        is_nested_call = 0
        profile_t1 = 0
        meth_name = 0x10865d70 "prepare"
        ima = 0x10865c00
        ima_flags = <value optimized out>
        imp_xxh = 0x15594520
        imp_msv = 0x15a64c36
        qsv = 0x58
#22 0x0000003bc2e90a96 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#23 0x0000003bc2e8a33e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#24 0x0000003bc2e37e9c in perl_run () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#25 0x00000000004017bc in main ()
No symbol table info available.
(gdb) A debugging session is active.

        Inferior 1 [process 11842] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
Detaching from program: /usr/bin/perl, process 11842
kill process 11842

Regards

Duncan
-----Original Message-----
From: Tomas Doran [mailto:bobtfish at bobtfish.net]
Sent: 11 January 2011 11:16
To: The elegant MVC web framework
Subject: Re: [Catalyst] Zeus and Catalyst


On 10 Jan 2011, at 22:05, Duncan Garland wrote:
> Are there any known problems using two serving running Zeus with a
> load balancer, fastcgi, Catalyst and Oracle 10.2 on Red Hat?

None known :)

> Our servers lock up occasionally. Half a dozen times a day, which is
> often enough to cause real embarassment. We restart them by killing
> the fastcgi process.
>
> The symptom is that something causes Catalyst to start returning
> zero length pages.
>
> eg
>
> 77.242.199.1 - - [10/Jan/2011:20:22:22 +0000] "GET /fcgi/catalyst/
> xxxx/script/xxxx_fastcgi.pl/javascript/calculation_accept HTTP/1.1"
> 200 0 "http://www.xxxx.co.uk/fcgi/catalyst/xxxx/script/xxxx_fastcgi.pl/home
> " "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13)
> Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729)"
>
> Half a dozen calls return this, then the server locks up completely
> until the process is killed.
>

Can you define 'locks up completely' here - is it spinning at 100% CPU?

What does strace say the fcgi process is doing when it's locked up?
Also, what does gdb say the stack is?

Is this fcgi on a unix domain socket, or a tcp socket?

> It doesn't happen on our other servers which don't have load
> balancers. One of the first things we did was to force the load
> balancer to always use the same server. That didn't help.

What is the balancer doing for it's health checks? Maybe this is what
is making it sick for some reason? (I've seen cases (non-Catalyst)
where load balancers making half requests / slightly mad requests /
closing the socket before getting a response / other mad things have
caused massive lossage, so it's worth thinking about)

> Could it be jumping out of the error handling and actually returning
> zero content? I can't see where.

It could be crapping itself half way through error handling?

> Is there a module dependency which would cause this?

No, I don't think so, but try running the latest release of FCGI.pm to
be sure..

> Is it Zeus or fastcgi?

One of these - something I _have_ seen in the past is that some (oh
hai nginx!) web server's fcgi implementations can get real confused if
you output a load of stuff to the fcgi error channel..

So try running your fastcgi with --keeperr, see if that has any effect..

Also, try loading Devel::SimpleTrace - with --keeperr
and ::SimpleTrace then you're more likely to get helpful errors out
before it all locks up.

Cheers
t0m

P.S. Once you have --keeperr on, then trying this: http://use.perl.org/~jjore/journal/39319
  could be useful to get a perl level backtrace out..


_______________________________________________
List: Catalyst at lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/



More information about the Catalyst mailing list