Apache Crashes at 12:15 AM Every Time!

Louie55 · 11-12-2005, 11:34 AM

Hello,

I am having a problem with a RedHat 9.0 Webserver that I am running with Apache 2. Once in awhile (not every day), about every week or 2, apache will crash. The funny part is, every time it crashes, the first "Status Warning" e-mail message is sent to me at 12:15 AM. It is always exactly 12:15 AM.

The "Status Warning" message gives me this as why Apache could not be restarted:

Code:

Nov 12 00:11:00 bulldog httpd: httpd shutdown succeeded
Nov 12 00:11:00 bulldog httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Nov 12 00:11:00 bulldog httpd: no listening sockets available, shutting down
Nov 12 00:11:00 bulldog httpd: Unable to open logs
Nov 12 00:11:00 bulldog httpd: httpd startup failed
Nov 12 00:11:00 bulldog httpd: httpd shutdown failed
Nov 12 00:11:00 bulldog httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Nov 12 00:11:00 bulldog httpd: no listening sockets available, shutting down
Nov 12 00:11:00 bulldog httpd: Unable to open logs
Nov 12 00:11:00 bulldog httpd: httpd startup failed
Nov 12 00:15:00 bulldog httpd: httpd shutdown failed
Nov 12 00:15:01 bulldog httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Nov 12 00:15:01 bulldog httpd: no listening sockets available, shutting down
Nov 12 00:15:01 bulldog httpd: Unable to open logs
Nov 12 00:15:01 bulldog httpd: httpd startup failed

It is always fixed by somebody else, not me, so I can't tell you exactly what the state of the machine is when it happens. They usually fix it by rebooting the server.

I have checked root crontab and nothing is set to run around 12:15 AM, so I can't pinpoint it. Any more ideas??

Thanks.

Louie

unSpawn · 11-12-2005, 05:08 PM

Seems there is already something listening on the HTTPS port when it tries to restart. Find out what it is.
Next to that check if there's any app that tries to restart httpd if it fails.

farslayer · 11-12-2005, 07:20 PM

Next time the server crashes, Try using netstat to determine what has the port open for listening.

netstat -alp | grep :https
or
netstat -alp | grep :443

I don't have a web server running on this box, but Here's what the output looks like for IPP (Internet priting protocol..) notice since I used the -p flag it lists the process name that has the port open ? in this case it's the cups printing daemon. you should see a similar listig for port 443 or https when you use this command.

Quote:

debian:~# netstat -alp | grep :ipp
tcp 0 0 *:ipp *:* LISTEN 9793/cupsd
udp 0 0 *:ipp *:* 9793/cupsd

KimVette · 11-12-2005, 08:58 PM

If the time is the same for every single occurence there are several possibilities:

- your apache build or one of your web apps has a time-sensitive bug
- a cron job is messing things up (check crontab, /etc/cron.*/, etc)
- Someone else's box infected with worms or a trojan horse is hitting your site with malformed or malicious requests (check the logs, such things would be obvious)

There are other possibilities but check for those first.

Louie55 · 11-13-2005, 07:43 PM

Quote:

Originally posted by unSpawn
Seems there is already something listening on the HTTPS port when it tries to restart. Find out what it is.
Next to that check if there's any app that tries to restart httpd if it fails.

I am about 99% sure that it is Apache that has port 443 bound up. What is was doing last time I SSH'ed into it when it was crashed, there were apache processes still running (many of them, no childs, only parent processes). They would not die in the regular way however. Lets say that one of the processes had a process ID of 873 just for example. If you do "kill -HUP 873", it has no effect, if you do "kill 873", that has no effect either. If you do "killall httpd" or "killall apache", that also has no effect as it comes back saying it can't find any. However, if I do "kill -KILL 873", that finally kills it. I am not sure what the -KILL does, I just found it on the internet somewhere. After all apache processes were finally killed, then apache came back up within 1 minute.

Now, to the question about what apps try to restart apache if it is down, there are 2 of them on the server. The first one is the DirectAdmin Web Hosting Control Panel Software, and the other is a system monitoring program (I can't remember the name). Both of these monitor services on the machine and attempt to restart them when they find one down. But, since apache is still running in a semi-crashed state, it can not be restarted because the crashed form of apache is still bound to the ports.

I had not thought of some other machine sending some kind of requests to crash it, I will check the logs.

Louie