[SOLVED] Apache2 Segmentation Faults Not Creating Core Dumps

Subject16 · 04-22-2011, 09:46 AM

Hello all.

I searched the forum because there appears to be no shortage of segmentation fault queries, but I haven't been able to figure this out with those resources, and every case is a little different anyway so I'm hoping someone will be able to shed some light on my problem.

I have a webserver that serves up multiple virtual hosts. I'm being told by users that they are intermittently unable to access their website; they get a white screen or they get a 404 error of some kind. I checked the logs and I see a ton of the following:

Code:

[notice] child pid 23838 exit signal Segmentation fault (11)

These happen pretty consistently every day during business hours, when the server is getting its heaviest use. There are no other errors or segfault instances being logged in syslog at all.

Here's the system setup:

Code:

Ubuntu 10.04.2 LTS, 2.6.32-28-server
Apache/2.2.14 (Ubuntu)
Compiled in modules:
  core.c
  mod_log_config.c
  mod_logio.c
  prefork.c
  http_core.c
  mod_so.c
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 jk_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 status_module (shared)

I set up the system to create core dumps following a combination of http://www.cyberciti.biz/tips/config...core-dump.html and http://httpd.apache.org/dev/debugging.html.

ulimit as my logged in user (not root):

Code:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 10000
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Contents of /etc/security/limits.conf:

Code:

*               soft    core            100000
root            hard    core            100000
*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50

I verified that there are no overriding ulimit settings in any of the shell configuration files, and I actually added "ulimit -c unlimited" to /etc/profile. I added CoreDumpDirectory /tmp/ to both my apache2.conf and httpd.conf. /tmp is 777. I've also changed /proc/sys/fs/suid_dumpable from 0 to 2 as per a suggestion in this forum. I've restarted apache, and yet I have no core dumps being created. I've searched the system using "sudo find / -name core*" and found nothing.

My understanding is that the only way to find out why these child processes are crashing, without actually having to recompile Apache to add in special switches, is to do a trace or analysis of the core dump files, but I'm not getting any.

stickman · 04-22-2011, 09:37 PM

Is your content pulling from some resource outside of Apache? For example, like from an Oracle client? It may be those processes that are seg faulting.

Subject16 · 04-25-2011, 10:10 AM

The content being served are PHP pages. I've read that when Apache segfaults it's usually due to a third party module, and even more so usually due to mod_php, but since the pages need that module I can't just pull it out to test. I was hoping the core dumps would give me an exact indication of where it was happening, if mod_php was indeed the culprit.

Subject16 · 04-25-2011, 12:18 PM

I figured out how to get the coredump, I think. I imaged this server prior to putting it into production so I used that to create a virtual machine. I then searched Google Groups for the error I reported above and happened across a post that pointed to a document within Apache itself that outlined how to enable core dumps. The link, if anyone could use it, is http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=368315 and the file in question is under /usr/share/doc/apache2.2-common/README.backtrace. Here it outlines steps needed to create a core dump, and sure enough it worked. The document specified a particular directory to use that I guess Apache created, /var/cache/apache2. Once I put that in and manually killed one of my child processes using kill -11, it created a core dump file for me to backtrace. I can only assume that this should work for my production server.

Hope this helps someone else.