OK, so im at my whits end with this. My web server keeps crapping out with the following error showing up in the error_log:
[Fri Feb 20 19:05:15 2009] [notice] child pid 4385 exit signal Segmentation fault (11)
Causes just blank pages when anything is requested.
Its frustrating because I know this is not my PHP code, I can run load tests all day and not get the error, but let the server run for a random amount of days then BOOM! SEG FAULT 11

Anyway, if you think you can help:

I have a 1 year old dell server with the following config:
PowerEdge 1950 III with 2x Quad Core procs at 2.33GHZ w/4GB Ram

Its ONLY function is to run as a LAMP server on OPENSUSE 11

Here are some details:

-uname -a
Linux server1 2.6.25.20-0.1-default #1 SMP 2008-12-12 20:30:38 +0100 x86_64 x86_64 x86_64 GNU/Linux

httpd2 -v
Server version: Apache/2.2.8 (Linux/SUSE)
Server built: Nov 23 2008 23:31:28

PHP VERSION
php -v
PHP 5.2.6 with Suhosin-Patch 0.9.6.2 (cli) (built: Aug 15 2008 14:53:30)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
with eAccelerator v0.9.5.3, Copyright (c) 2004-2006 eAccelerator, by eAccelerator
with Suhosin v0.9.24, Copyright (c) 2007, by SektionEins GmbH

Here are the modules I have loaded:

APACHE_MODULES="authz_host alias dir log_config mime setenvif expires include deflate headers php5"

I have ran without eaccelerator too and get the same results. Memory has been replaced with new modules too. Thanks!

I'm definitely no expert on Apache. But I assume that this process id belongs to some real program. Can you monitor this closely and figure out which program is actually crashing? (Or do you already know?) "ps -e" gives the list of currently running process-ids/programs.

Also, how do you get it working again? Does restarting apache (temporarily) fix the problem?

Again, I'm no Apache expert, but a seg fault basically means that some program tried to access memory that it doesn't own. (This is often the result of a buffer overrun -- where the program tries to stuff too much data into an array.) This is a result of bad programming. So if you can find the offending program, then you might learn what the problem is.

Eh, you probably knew that already, but hope it helps a little.

Bad RAM could also cause such crashes. Try stressing it with something like memtest86 and see if you can induce a crash.

I appreciate the quick responses from you guys. Thanks! The process IDs are pretty random and end with the seg fault:


[Sat Feb 21 11:32:11 2009] [notice] child pid 10205 exit signal Segmentation fault (11)
[Sat Feb 21 11:40:28 2009] [notice] child pid 10285 exit signal Segmentation fault (11)
[Sat Feb 21 11:40:28 2009] [notice] child pid 10578 exit signal Segmentation fault (11)
[Sat Feb 21 11:40:28 2009] [notice] child pid 10579 exit signal Segmentation fault (11)
[Sat Feb 21 11:40:41 2009] [notice] child pid 10580 exit signal Segmentation fault (11)
[Sat Feb 21 11:43:34 2009] [notice] child pid 10582 exit signal Segmentation fault (11)
[Sat Feb 21 11:45:04 2009] [notice] child pid 10583 exit signal Segmentation fault (11)

Im not an Apache expert either but i think pre-fork generates these processes to handle requests. I can get these things all day long until i restart apache. I do have a warranty on the server and hardware is covered. I will run memtest86 on this bad boy and get back to you with the results. I searched the forum before posting and found lots on SEG FAULTs, but really nothing similar to what I have experienced here. Frustrating!

OK, I can confidently say that the hardware is definitely not the issue here. I ran full dell diagnostics on the system for 3 hours as recommended by Dell tech support. Every test passed several times. I ran memtest86 overnight with no errors found. This has got to be a software issue. Any ideas anyone?