Apache randomly crashes
 

This has been happening for a while now and I can't seem to isolate the issue. Apache will run fine for anywhere from a few days to a few weeks but then all of a sudden it will "crash". I put that in quotes because the process is not completely crashing. Basically when this happens, Apache just refuses to accept any connections and cannot be restarted using the init script (it simply fails to stop then fails to start and gives an error about the address being in use). Every time this happens, I just do a "killall httpd", then start the process with the init script again and everything works perfectly. Of course having this happen is quite a nuisance.

Does anyone know what could be going on here? I have checked the error log and it only contains notices, no errors.

I'd appreciate any help. Thanks

More info needed to troubleshoot. You can make Apache log better chatter with the directive:
LogLevel info

After you've added that to your config:
Next time Apache runs into problems, check your error logs again.

Ok I did that. But in the mean time, is it possible that it is reaching the max connections or would that have shown up in my error log? I had the log level set to warn.

Sure - but a number of things are possible. Let's see what Apache himself tells us.

I can't seem to get any info from the error logs. My error logs for the past 8+ days are non existent and before that, I do not see any useful information. Here is the message I get when I try to restart apache after a "crash".

[root@srv1 httpd]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@srv1 httpd]#

It looks like something else is listening on port 80, so apache can't start. What's running on 80?

You might want to get a look at netstat when apache is 'crashed'. Do something like:

netstat -an | grep :80

You can add a wc -l to the end to see how many connections there are. If it's around your max clients configured in apache, that might be the issue.

Hi,



May be your apache didn't stop properly and it makes you to kill all the httpd processes to restart it. I had the same issue with my client's server, and the reason for that was the server reached maximum clients and got hanged. You can check them with your error logs, while analyzing I found that developer made a mistake while developing a ffmpeg script and that ffmpeg process forms a loop and multiplies the number of httpd requests to the server which completely eats up the number of connections.

Doesn't look like anything else is running on that port. If I was reaching the max connections, it would seem that would show up in the error log.

I suppose it's possible that a module is causing this. How would I go about finding the module that was causing this?

I'm still having this issue. I have tried everything I can think of, including changing the apache LogLevel to debug. Even in debug, there is nothing in the error log or system messages log around the time apache "crashes".

Does anyone have any ideas?

Does it still say that it cannot bind to port 80?

Could you give us the out put of the following command:
netstat -pant|grep ":80 "

Here is my apache error log with debug turned on. Keep in mind that I have no idea if these messages are from around the time of the crash because I do not know when it crashed.

Here is the output of that netstat command while apache is crashed

Crashed is not the same as "no longer accepting connections"
Try running strace -p 2460

you might also try lsof |grep "^httpd"

But also consider that it is not stopping properly with the init script after a "crash"

Here is some output from strace (not sure what this does, but it seemed to be never ending so these are the last few lines of output if that helps at all). From what I could tell, it was almost the same thing repeated over and over again.


As for lsof:

Thanks

I don't have an answer, I don't know centos. I would take that POLLHUP info and search the apache maillist.
Are you doing a graceful shutdown? If I recall the graceful shutdown waits until some conditions are met.
Possibly some sockets are misbehaving.

I see you have selinux too, wheter or not it's a culprit here I don't know. It can get in the way sometimes.

Well, I didn't try to do any type of shutdown until it was already crashed. In the error log there is a line indicating a graceful restart

[Sun Nov 27 04:02:17 2011] [notice] Graceful restart requested, doing restart

Is it possible that apache is doing graceful restarts on its own and maybe the conditions are not being met so the process becomes halted? In that case wouldn't it at least log the request?

All I really know is that I started having this issue about a year ago and cannot for the life of me figure out what is causing it. I've been using apache for a number of years, and every other issue I've had has been indicated in the log and relatively straightforward to resolve.