LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 11-13-2009, 04:07 AM   #1
RattleSn@ke
Member
 
Registered: Oct 2007
Location: Netherlands, ZH
Posts: 32

Rep: Reputation: 15
Debug apache with core dumps


Hi all,

I have the following issue. We have 7 webservers with apache / PHP5. Every day, without any clear reason all servers are getting high loads and all Apache children die. I found these messages in the Apache error log:
[Wed Nov 11 14:39:22 2009] [notice] child pid 12849 exit signal Segmentation fault (11)
[Wed Nov 11 14:39:23 2009] [notice] child pid 12556 exit signal Segmentation fault (11)
zend_mm_heap corrupted
[Wed Nov 11 14:39:25 2009] [notice] child pid 12871 exit signal Segmentation fault (11)
zend_mm_heap corrupted
zend_mm_heap corrupted

So I changed the Apache config so it would dump coredumps in /tmp. Then I waited 'till the next major crash. After a few hours the first crash occurred and in one run I had 1,5G of coredumps.

So I wanted to look inside the coredump and executed: gdb /usr/sbin/httpd core.12642 and did a 'bt full', but I can't make anything out of it:
Code:
# gdb /usr/sbin/httpd core.12642
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
(no debugging symbols found)
Loaded symbols for /usr/sbin/httpd
Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /usr/lib/libpcreposix.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcreposix.so.0
Reading symbols from /usr/lib/libaprutil-0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libaprutil-0.so.0
Reading symbols from /usr/lib/libldap-2.2.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap-2.2.so.7
Reading symbols from /usr/lib/liblber-2.2.so.7...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.2.so.7
Reading symbols from /lib/tls/i686/libdb-4.2.so...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/i686/libdb-4.2.so
Reading symbols from /usr/lib/libexpat.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libexpat.so.0
Reading symbols from /usr/lib/libapr-0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libapr-0.so.0
Reading symbols from /lib/tls/librt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libdl.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libssl.so.4...
(no debugging symbols found)...done.
Loaded symbols for /lib/libssl.so.4
Reading symbols from /lib/libcrypto.so.4...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/httpd/modules/mod_access.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_access.so
Reading symbols from /usr/lib/httpd/modules/mod_auth.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_anon.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_anon.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_dbm.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_dbm.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_digest.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_digest.so
Reading symbols from /usr/lib/httpd/modules/mod_ldap.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_ldap.so
Reading symbols from /usr/lib/httpd/modules/mod_auth_ldap.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_ldap.so
Reading symbols from /usr/lib/httpd/modules/mod_include.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_include.so
Reading symbols from /usr/lib/httpd/modules/mod_log_config.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_log_config.so
Reading symbols from /usr/lib/httpd/modules/mod_env.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_env.so
Reading symbols from /usr/lib/httpd/modules/mod_mime_magic.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime_magic.so
Reading symbols from /usr/lib/httpd/modules/mod_cern_meta.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cern_meta.so
Reading symbols from /usr/lib/httpd/modules/mod_expires.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_expires.so
Reading symbols from /usr/lib/httpd/modules/mod_deflate.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_deflate.so
Reading symbols from /usr/lib/httpd/modules/mod_headers.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_headers.so
Reading symbols from /usr/lib/httpd/modules/mod_usertrack.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_usertrack.so
Reading symbols from /usr/lib/httpd/modules/mod_setenvif.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_setenvif.so
Reading symbols from /usr/lib/httpd/modules/mod_mime.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime.so
Reading symbols from /usr/lib/httpd/modules/mod_dav.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav.so
Reading symbols from /usr/lib/httpd/modules/mod_status.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_status.so
Reading symbols from /usr/lib/httpd/modules/mod_autoindex.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_autoindex.so
Reading symbols from /usr/lib/httpd/modules/mod_asis.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_asis.so
Reading symbols from /usr/lib/httpd/modules/mod_info.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_info.so
Reading symbols from /usr/lib/httpd/modules/mod_dav_fs.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav_fs.so
Reading symbols from /usr/lib/httpd/modules/mod_vhost_alias.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so
Reading symbols from /usr/lib/httpd/modules/mod_negotiation.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_negotiation.so
Reading symbols from /usr/lib/httpd/modules/mod_dir.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_dir.so
Reading symbols from /usr/lib/httpd/modules/mod_imap.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_imap.so
Reading symbols from /usr/lib/httpd/modules/mod_actions.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_actions.so
Reading symbols from /usr/lib/httpd/modules/mod_speling.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_speling.so
Reading symbols from /usr/lib/httpd/modules/mod_userdir.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_userdir.so
Reading symbols from /usr/lib/httpd/modules/mod_alias.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_alias.so
Reading symbols from /usr/lib/httpd/modules/mod_rewrite.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_ftp.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_http.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
Reading symbols from /usr/lib/httpd/modules/mod_proxy_connect.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
Reading symbols from /usr/lib/httpd/modules/mod_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_suexec.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_suexec.so
Reading symbols from /usr/lib/httpd/modules/mod_disk_cache.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_file_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_file_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_mem_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mem_cache.so
Reading symbols from /usr/lib/httpd/modules/mod_cgi.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /usr/lib/httpd/modules/mod_version.so...
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_version.so
Reading symbols from /usr/lib/httpd/modules/libphp5.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/libphp5.so
Reading symbols from /usr/lib/sse2/libgmp.so.3...done.
Loaded symbols for /usr/lib/sse2/libgmp.so.3
Reading symbols from /usr/lib/libbz2.so.1...done.
Loaded symbols for /usr/lib/libbz2.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /usr/lib/php/modules/curl.so...done.
Loaded symbols for /usr/lib/php/modules/curl.so
Reading symbols from /usr/lib/libcurl.so.3...done.
Loaded symbols for /usr/lib/libcurl.so.3
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/php/modules/dbase.so...done.
Loaded symbols for /usr/lib/php/modules/dbase.so
Reading symbols from /usr/lib/php/modules/dom.so...done.
Loaded symbols for /usr/lib/php/modules/dom.so
Reading symbols from /usr/lib/php/modules/gd.so...done.
Loaded symbols for /usr/lib/php/modules/gd.so
Reading symbols from /usr/lib/libt1.so.5...done.
Loaded symbols for /usr/lib/libt1.so.5
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/X11R6/lib/libX11.so.6...done.
Loaded symbols for /usr/X11R6/lib/libX11.so.6
Reading symbols from /usr/X11R6/lib/libXpm.so.4...done.
Loaded symbols for /usr/X11R6/lib/libXpm.so.4
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /usr/X11R6/lib/libXext.so.6...done.
Loaded symbols for /usr/X11R6/lib/libXext.so.6
Reading symbols from /usr/lib/php/modules/json.so...done.
Loaded symbols for /usr/lib/php/modules/json.so
Reading symbols from /usr/lib/php/modules/mysql.so...done.
Loaded symbols for /usr/lib/php/modules/mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.14...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.14
Reading symbols from /usr/lib/php/modules/mysqli.so...done.
Loaded symbols for /usr/lib/php/modules/mysqli.so
Reading symbols from /usr/lib/php/modules/pdo.so...done.
Loaded symbols for /usr/lib/php/modules/pdo.so
Reading symbols from /usr/lib/php/modules/pdo_mysql.so...done.
Loaded symbols for /usr/lib/php/modules/pdo_mysql.so
Reading symbols from /usr/lib/php/modules/pdo_sqlite.so...done.
Loaded symbols for /usr/lib/php/modules/pdo_sqlite.so
Reading symbols from /usr/lib/libsqlite3.so.0...done.
Loaded symbols for /usr/lib/libsqlite3.so.0
Reading symbols from /usr/lib/php/modules/soap.so...done.
Loaded symbols for /usr/lib/php/modules/soap.so
Reading symbols from /usr/lib/php/modules/wddx.so...done.
Loaded symbols for /usr/lib/php/modules/wddx.so
Reading symbols from /usr/lib/php/modules/xmlreader.so...done.
Loaded symbols for /usr/lib/php/modules/xmlreader.so
Reading symbols from /usr/lib/php/modules/xmlwriter.so...done.
Loaded symbols for /usr/lib/php/modules/xmlwriter.so
Reading symbols from /usr/lib/php/modules/xsl.so...done.
Loaded symbols for /usr/lib/php/modules/xsl.so
Reading symbols from /usr/lib/libexslt.so.0...done.
Loaded symbols for /usr/lib/libexslt.so.0
Reading symbols from /usr/lib/libxslt.so.1...done.
Loaded symbols for /usr/lib/libxslt.so.1
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/php/modules/zip.so...done.
Loaded symbols for /usr/lib/php/modules/zip.so
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
#0  0x010dd99b in ini_lex ()
   from /etc/httpd/modules/libphp5.so
(gdb) bt full
#0  0x010dd99b in ini_lex () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#1  0x01105b92 in zend_hash_destroy () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#2  0x010dfeee in shutdown_compiler () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#3  0x010fa4c6 in zend_deactivate () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#4  0x010b441b in php_request_shutdown () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#5  0x01199566 in zend_get_zval_ptr_ptr () from /etc/httpd/modules/libphp5.so
No symbol table info available.
#6  0x00baca07 in ap_run_handler () from /usr/sbin/httpd
No symbol table info available.
#7  0x00bace73 in ap_invoke_handler () from /usr/sbin/httpd
No symbol table info available.
#8  0x00ba98f5 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#9  0x00ba463f in _start () from /usr/sbin/httpd
No symbol table info available.
#10 0x0839b268 in ?? ()
No symbol table info available.
#11 0x00000004 in ?? ()
No symbol table info available.
#12 0x0839b268 in ?? ()
No symbol table info available.
#13 0x0837efd0 in ?? ()
No symbol table info available.
#14 0x0837efe4 in ?? ()
No symbol table info available.
#15 0x0837eef8 in ?? ()
No symbol table info available.
#16 0x00000001 in ?? ()
No symbol table info available.
#17 0x00bd4414 in ?? () from /usr/sbin/httpd
No symbol table info available.
#18 0x08209278 in ?? ()
No symbol table info available.
#19 0x00000000 in ?? ()
No symbol table info available.
(gdb)
Can someone help me a bit with debugging? I know it's a pain but any directions would be helpful.

Thanks in advance!
Onno.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How do I redirect core dumps? felbvts Red Hat 4 06-30-2009 12:17 AM
RPM Dumps Core chingkhuba Red Hat 6 04-15-2009 03:03 AM
Core dumps amitktd Solaris / OpenSolaris 2 05-31-2007 01:39 AM
Turning on Core dumps lostinvietnam Linux - Kernel 3 02-09-2007 02:15 AM
Ps dumps core in linux 2.4.22 rbecker Linux - General 2 01-04-2006 12:43 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 06:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration