Hi all,
I have the following issue. We have 7 webservers with apache / PHP5. Every day, without any clear reason all servers are getting high loads and all Apache children die. I found these messages in the Apache error log:
[Wed Nov 11 14:39:22 2009] [notice] child pid 12849 exit signal Segmentation fault (11)
[Wed Nov 11 14:39:23 2009] [notice] child pid 12556 exit signal Segmentation fault (11)
zend_mm_heap corrupted
[Wed Nov 11 14:39:25 2009] [notice] child pid 12871 exit signal Segmentation fault (11)
zend_mm_heap corrupted
zend_mm_heap corrupted
So I changed the Apache config so it would dump coredumps in /tmp. Then I waited 'till the next major crash. After a few hours the first crash occurred and in one run I had 1,5G of coredumps.
So I wanted to look inside the coredump and executed: gdb /usr/sbin/httpd core.12642 and did a 'bt full', but I can't make anything out of it:
Code:
# gdb /usr/sbin/httpd core.12642
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
(no debugging symbols found)
Loaded symbols for /usr/sbin/httpd
Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /usr/lib/libpcreposix.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcreposix.so.0
Reading symbols from /usr/lib/libaprutil-0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libaprutil-0.so.0
Reading symbols from /usr/lib/libldap-2.2.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap-2.2.so.7
Reading symbols from /usr/lib/liblber-2.2.so.7...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.2.so.7
Reading symbols from /lib/tls/i686/libdb-4.2.so...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/i686/libdb-4.2.so
Reading symbols from /usr/lib/libexpat.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libexpat.so.0
Reading symbols from /usr/lib/libapr-0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libapr-0.so.0
Reading symbols from /lib/tls/librt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libdl.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libssl.so.4...
(no debugging symbols found)...done.
Loaded symbols for /lib/libssl.so.4
Reading symbols from /lib/libcrypto.so.4...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/httpd/modules/mod_access.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_access.so
Reading symbols from /usr/lib/httpd/modules/mod_auth.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_anon.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_anon.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_dbm.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_dbm.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_digest.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_digest.so
Reading symbols from /usr/lib/httpd/modules/mod_ldap.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_ldap.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_ldap.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_ldap.so
Reading symbols from /usr/lib/httpd/modules/mod_include.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_include.so
Reading symbols from /usr/lib/httpd/modules/mod_log_config.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_log_config.so
Reading symbols from /usr/lib/httpd/modules/mod_env.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_env.so
Reading symbols from /usr/lib/httpd/modules/mod_mime_magic.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime_magic.so
Reading symbols from /usr/lib/httpd/modules/mod_cern_meta.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cern_meta.so
Reading symbols from /usr/lib/httpd/modules/mod_expires.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_expires.so
Reading symbols from /usr/lib/httpd/modules/mod_deflate.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_deflate.so
Reading symbols from /usr/lib/httpd/modules/mod_headers.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_headers.so
Reading symbols from /usr/lib/httpd/modules/mod_usertrack.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_usertrack.so
Reading symbols from /usr/lib/httpd/modules/mod_setenvif.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_setenvif.so
Reading symbols from /usr/lib/httpd/modules/mod_mime.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime.so
Reading symbols from /usr/lib/httpd/modules/mod_dav.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav.so
Reading symbols from /usr/lib/httpd/modules/mod_status.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_status.so
Reading symbols from /usr/lib/httpd/modules/mod_autoindex.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_autoindex.so
Reading symbols from /usr/lib/httpd/modules/mod_asis.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_asis.so
Reading symbols from /usr/lib/httpd/modules/mod_info.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_info.so
Reading symbols from /usr/lib/httpd/modules/mod_dav_fs.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav_fs.so
Reading symbols from /usr/lib/httpd/modules/mod_vhost_alias.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so
Reading symbols from /usr/lib/httpd/modules/mod_negotiation.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_negotiation.so
Reading symbols from /usr/lib/httpd/modules/mod_dir.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dir.so
Reading symbols from /usr/lib/httpd/modules/mod_imap.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_imap.so
Reading symbols from /usr/lib/httpd/modules/mod_actions.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_actions.so
Reading symbols from /usr/lib/httpd/modules/mod_speling.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_speling.so
Reading symbols from /usr/lib/httpd/modules/mod_userdir.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_userdir.so
Reading symbols from /usr/lib/httpd/modules/mod_alias.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_alias.so
Reading symbols from /usr/lib/httpd/modules/mod_rewrite.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_ftp.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_http.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_connect.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
Reading symbols from /usr/lib/httpd/modules/mod_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_suexec.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_suexec.so
Reading symbols from /usr/lib/httpd/modules/mod_disk_cache.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_file_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_file_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_mem_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mem_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_cgi.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /usr/lib/httpd/modules/mod_version.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_version.so
Reading symbols from /usr/lib/httpd/modules/libphp5.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/libphp5.so
Reading symbols from /usr/lib/sse2/libgmp.so.3...done.
Loaded symbols for /usr/lib/sse2/libgmp.so.3
Reading symbols from /usr/lib/libbz2.so.1...done.
Loaded symbols for /usr/lib/libbz2.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /usr/lib/php/modules/curl.so...done.
Loaded symbols for /usr/lib/php/modules/curl.so
Reading symbols from /usr/lib/libcurl.so.3...done.
Loaded symbols for /usr/lib/libcurl.so.3
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/php/modules/dbase.so...done.
Loaded symbols for /usr/lib/php/modules/dbase.so
Reading symbols from /usr/lib/php/modules/dom.so...done.
Loaded symbols for /usr/lib/php/modules/dom.so
Reading symbols from /usr/lib/php/modules/gd.so...done.
Loaded symbols for /usr/lib/php/modules/gd.so
Reading symbols from /usr/lib/libt1.so.5...done.
Loaded symbols for /usr/lib/libt1.so.5
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/X11R6/lib/libX11.so.6...done.
Loaded symbols for /usr/X11R6/lib/libX11.so.6
Reading symbols from /usr/X11R6/lib/libXpm.so.4...done.
Loaded symbols for /usr/X11R6/lib/libXpm.so.4
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /usr/X11R6/lib/libXext.so.6...done.
Loaded symbols for /usr/X11R6/lib/libXext.so.6
Reading symbols from /usr/lib/php/modules/json.so...done.
Loaded symbols for /usr/lib/php/modules/json.so
Reading symbols from /usr/lib/php/modules/mysql.so...done.
Loaded symbols for /usr/lib/php/modules/mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.14...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.14
Reading symbols from /usr/lib/php/modules/mysqli.so...done.
Loaded symbols for /usr/lib/php/modules/mysqli.so
Reading symbols from /usr/lib/php/modules/pdo.so...done.
Loaded symbols for /usr/lib/php/modules/pdo.so
Reading symbols from /usr/lib/php/modules/pdo_mysql.so...done.
Loaded symbols for /usr/lib/php/modules/pdo_mysql.so
Reading symbols from /usr/lib/php/modules/pdo_sqlite.so...done.
Loaded symbols for /usr/lib/php/modules/pdo_sqlite.so
Reading symbols from /usr/lib/libsqlite3.so.0...done.
Loaded symbols for /usr/lib/libsqlite3.so.0
Reading symbols from /usr/lib/php/modules/soap.so...done.
Loaded symbols for /usr/lib/php/modules/soap.so
Reading symbols from /usr/lib/php/modules/wddx.so...done.
Loaded symbols for /usr/lib/php/modules/wddx.so
Reading symbols from /usr/lib/php/modules/xmlreader.so...done.
Loaded symbols for /usr/lib/php/modules/xmlreader.so
Reading symbols from /usr/lib/php/modules/xmlwriter.so...done.
Loaded symbols for /usr/lib/php/modules/xmlwriter.so
Reading symbols from /usr/lib/php/modules/xsl.so...done.
Loaded symbols for /usr/lib/php/modules/xsl.so
Reading symbols from /usr/lib/libexslt.so.0...done.
Loaded symbols for /usr/lib/libexslt.so.0
Reading symbols from /usr/lib/libxslt.so.1...done.
Loaded symbols for /usr/lib/libxslt.so.1
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/php/modules/zip.so...done.
Loaded symbols for /usr/lib/php/modules/zip.so
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
#0 0x010dd99b in ini_lex ()
from /etc/httpd/modules/libphp5.so
(gdb) bt full
#0 0x010dd99b in ini_lex () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#1 0x01105b92 in zend_hash_destroy () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#2 0x010dfeee in shutdown_compiler () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#3 0x010fa4c6 in zend_deactivate () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#4 0x010b441b in php_request_shutdown () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#5 0x01199566 in zend_get_zval_ptr_ptr () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#6 0x00baca07 in ap_run_handler () from /usr/sbin/httpd
No symbol table info available.
#7 0x00bace73 in ap_invoke_handler () from /usr/sbin/httpd
No symbol table info available.
#8 0x00ba98f5 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#9 0x00ba463f in _start () from /usr/sbin/httpd
No symbol table info available.
#10 0x0839b268 in ?? ()
No symbol table info available.
#11 0x00000004 in ?? ()
No symbol table info available.
#12 0x0839b268 in ?? ()
No symbol table info available.
#13 0x0837efd0 in ?? ()
No symbol table info available.
#14 0x0837efe4 in ?? ()
No symbol table info available.
#15 0x0837eef8 in ?? ()
No symbol table info available.
#16 0x00000001 in ?? ()
No symbol table info available.
#17 0x00bd4414 in ?? () from /usr/sbin/httpd
No symbol table info available.
#18 0x08209278 in ?? ()
No symbol table info available.
#19 0x00000000 in ?? ()
No symbol table info available.
(gdb)
Can someone help me a bit with debugging? I know it's a pain but any directions would be helpful.
Thanks in advance!
Onno.