apache 2.2.x dying several times an hour

dazdaz · 05-27-2011, 04:53 PM

Hi,

I manage an Apache 2.2.3 web server. It stops working several times an hour, so I removed the semaphores and restarted it as a workaround, but I dont know what is causing this. Lately i've noticed that the workaround is not working at all.

Code:

0 */1 * * * /usr/bin/ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'
0 */1 * * * /sbin/service httpd restart 1>/dev/null

Below is what I can see in my error_log file.

What is the mutex message telling me. To be honest, I don't know much about semaphores nor how they work. You can see after the semaphore issue, that Apache is sent the SIGTERM signal.

I have seen other posts on this topic, but not explicitly with Linux 2.6.22 and Apache 2.2.3 with a description of the issue and a permanent fix.

Code:

[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:04 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Fri May 27 22:00:08 2011] [notice] caught SIGTERM, shutting down
[Fri May 27 22:00:17 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri May 27 22:00:18 2011] [notice] Digest: generating secret for digest authentication ...
[Fri May 27 22:00:18 2011] [notice] Digest: done
PHP Warning:  PHP Startup: eAccelerator: Unable to initialize module\nModule compiled with module API=20050922, debug=0, thread-safety=0\nPHP    compiled with module API=20060613, debug=0, thread-safety=0\nThese options need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: fileinfo: Unable to initialize module\nModule compiled with module API=20050922, debug=0, thread-safety=0\nPHP    compiled with module API=20060613, debug=0, thread-safety=0\nThese options need to match\n in Unknown on line 0

tronayne · 05-28-2011, 08:22 AM

There is a section in the Apache 2.21.x manual that speaks to this: http://httpd.apache.org/docs/2.2/misc/perf-tuning.html. Particularly look for Compile-Time Configuration Issues and give it a careful read (taking note that the accept Serialization - multiple sockets may or may not be relevant. Much of the discussion centers around sockets and mutex function (and your hardware and IPC configuration).

It could be as simple as that you do not have enough shared memory allocated; if you were to execute df -h | grep shm you should see something on the order of

Code:

tmpfs           3.9G     0  3.9G   0% /dev/shm

If you don't, you may need to look into allocating more.

Also, take a hard look at your httpd.conf file; if you can, make sure the status module is loaded and allow server status reports to be generated by mod_status and enable ExtendedStatus reports (might give you some hints). I'll be happy to send or post my httpd.conf file if that would help, but be aware that Slackware does the HTTPD configuration in /etc/httpd and includes optional configuration settings in /etc/httpd/extra which may be a little confusing if your implementation is not similar.

Hope this helps some.