We recently had to migrate servers from Redhat 7.3 to Redhat Enterprise. We are running a very high traffic site on a dual 2.4 GHZ 2 gig ram machine. Our software is PHP/Mysql intensive. Now most of our migration went relatively smoothly however we are running into problems with the apache 2 that came with Redhat Enterprise.

Every couple of hours our sites become all unreachable and after checking the error_log files we see this as the last thing to log:

[Thu Jul 15 16:52:57 2004] [notice] child pid 22527 exit signal Segmentation fault (11)
[Thu Jul 15 16:52:59 2004] [notice] child pid 22528 exit signal Segmentation fault (11)


I have tried tweaking the conf files as much as possible but can't get around this error. At the time this happened I happened to catch the server status information and it said 130 requests currently being processed and 0 idle workers.

Now 130 is the number of MaxClients we have setup in the conf file.

Any ideas on this would be very helpful.

Actually I am not sure if it is the seg faults that are choaking apache. It is wierd because the I just did a grep for Segmentation Fault of the error_logs and find seg faults every hour, however our server hasn't gone done for the last 5 hours (holding breath). Anything that would cause apache to just stop operating?

When it happens all the apache processes are still there, it is just you can't connect or pull any pages from it. So we basically have to run apachectl stop like 3 times to get it to die and finally restart. After the restart the server runs lightning fast for a good little while...

I have seen segmentation faults occur due to faulty (usually new) memory. Did you happen to upgrade memory also or shift to a new machine?

You can run memtest for a few hours and see if it turns up anything.
cheers.
Ken

Yeah we did just move machines so maybe that is the problem. I will try a memtest and see what it resolves. thanks.

Humm after watching this server for the last couple of days (and watching it die every so often) I am seeing some trends in the server-status. I notice that it doesn't seem to be finishing requests, or rather closing them. People are view the pages fine but it seems after that a few connections stay stuck or something... Currently we have:

204 requests currently being processed, 8 idle workers

Which is WAY higher than it should be for this time of day. Also I have noticed that in the list of processes that the top of the list doesn't seem to change, they just sit there stuck on Gracefully finishing but never finish... Has anyone know why this is happening and if this is possibly why my server just stops working after a while?

After further investigation we find that certain scripts of ours are not finishing and thus causing the httpd process it was working on to be left in a constant state of waiting. This process is then riddled useless and I think is the reason why our server eventually stop working (too many processes or connections or something). Anyway is there anyway to make this process just die, or timeout or something?

This is way over my head but perhaps you could (or somebody could) fashion some sort of Cron job that monitored the health of that process and killed it if it was frozen.