Apache and SSH seem to restart intermittently

dmrossi · 09-16-2009, 03:10 AM

I'm hosting my own site and I recently switched from Windows Server to Ubuntu as my hosting OS. I signed up for a free account from pingdom.com to monitor my site uptime. As I soon noticed, my server seemed to be going offline quite frequently and for varying amounts of time. (i.e. down for 12 minutes, up for 30 minutes, down for 42 minutes, ect.) When I received notice from pingdom that my website was offline, I attempted to connect to my site from a remote location and it was indeed, unreachable. Not only was my server not responding to remote connections on port 80, but it also failed to respond to my remote SSH attempts. However, when my site was up according to pingdom, I was able to connect via both ports from a remote location.

I've been making a point of specifying remote location for a reason. It was just today that I noticed that if my site was down remotely, accessing it locally "woke it up." After I accessed it locally, I was then able to access it remotely, via ports 80 and 22. Of course, after a while, my remote connections timed out again, as the services on my server seemed to have gone asleep again. When the services are non-responsive from remote locations, the server itself seems fine and responsive.

It's also important to mention that my router, ISP, and DNS servers are all not issues since I was able to host my site using Windows Server without this issue.

Any help would be greatly appreciated. I apologize in advance if I have omitted some very obvious and vital information as I am new to Linux. I also have searched Google and these forums, however I find it hard to phrase my problem in searchable terms. Thank you.

Meson · 09-16-2009, 09:45 AM

It sounds like too big of a coincidence that httpd and sshd are restarting randomaly, at the exact same time. I think you should be trying to diagnose some sort of networking issue. You should check out your log files (like daemon.log) to convince yourself that these services are not actually going down.

dmrossi · 09-16-2009, 10:58 AM

Daemon.log:

Code:

Sep 16 06:12:58 Server dhclient: DHCPREQUEST of 192.168.1.104 on eth0 to 192.168.1.1 port 67
Sep 16 06:12:58 Server dhclient: DHCPACK of 192.168.1.104 from 192.168.1.1
Sep 16 06:12:58 Server dhclient: bound to 192.168.1.104 -- renewal in 35579 seconds.

Networking issue? I had the same setup with Windows Server. Maybe I'm not sure what you mean by this.

chrism01 · 09-16-2009, 07:38 PM

Try /var/log/messages, /var/log/secure

Wim Sturkenboom · 09-16-2009, 10:54 PM

When apache restarts, it will write a message in a log file; on my Slackware 10.1 this is var/log/apache/error_log. So you can find there if it's actually restarted.

dmrossi · 09-17-2009, 12:07 AM

Messages (/var/log/messages):

Code:

Sep 14 12:32:42 Server -- MARK --
Sep 14 12:52:42 Server -- MARK --
Sep 14 13:12:42 Server -- MARK --
Sep 14 13:32:42 Server -- MARK --
Sep 14 13:52:42 Server -- MARK --
Sep 14 14:12:42 Server -- MARK --
Sep 14 14:32:42 Server -- MARK --
Sep 14 14:52:42 Server -- MARK --
Sep 14 15:12:42 Server -- MARK --

I do not have a secure log.

Apache Log:

Code:

[Sun Sep 13 07:59:23 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations
[Sun Sep 13 10:38:29 2009] [error] [client 205.209.161.42] Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Tue Sep 15 00:23:40 2009] [error] [client 212.241.213.73] client denied by server configuration: /var/www/.htpasswd
[Tue Sep 15 00:26:55 2009] [error] [client 212.241.213.73] client denied by server configuration: /var/www/.htpasswd
[Tue Sep 15 21:30:57 2009] [error] [client 77.92.68.146] Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Wed Sep 16 10:34:00 2009] [notice] caught SIGTERM, shutting down
[Wed Sep 16 10:34:45 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations

Wim Sturkenboom · 09-17-2009, 01:09 AM

Below my simulation where I manually stop apache.

Code:

[Thu Sep 17 08:06:03 2009] [notice] Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7d PHP/4.3.10 configured -- resuming normal operations
[Thu Sep 17 08:06:03 2009] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Thu Sep 17 08:06:27 2009] [notice] caught SIGTERM, shutting down
[Thu Sep 17 08:06:36 2009] [notice] Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7d PHP/4.3.10 configured -- resuming normal operations
[Thu Sep 17 08:06:36 2009] [notice] Accept mutex: sysvsem (Default: sysvsem)

You have the same message in your apache log. Something/somebody is restarting your apache.

dmrossi · 09-17-2009, 01:41 AM

But doesn't the line after it indicate that it is going back up again after 9 seconds in your case, 45 in mine?

Apache Log:

Code:

[Sun Sep 13 07:59:23 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations
[Sun Sep 13 10:38:29 2009] [error] [client 205.209.161.42] Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Tue Sep 15 00:23:40 2009] [error] [client 212.241.213.73] client denied by server configuration: /var/www/.htpasswd
[Tue Sep 15 00:26:55 2009] [error] [client 212.241.213.73] client denied by server configuration: /var/www/.htpasswd
[Tue Sep 15 21:30:57 2009] [error] [client 77.92.68.146] Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Wed Sep 16 10:34:00 2009] [notice] caught SIGTERM, shutting down
[Wed Sep 16 10:34:45 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations

SSH and Apache are definitely down for more than a minute at a time. Or does this next line not necessarily mean it is starting back up 45 seconds later?

Wim Sturkenboom · 09-17-2009, 02:13 AM

Yep, it indicates that. But check the full log and see if it's always like that.

The only valid reason for a sigterm that I can think of is if you use logrotate.

dmrossi · 09-17-2009, 02:26 AM

That was a full log, but here's another log in the same directory:

Code:

[Mon Sep 07 17:30:39 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations
[Mon Sep 07 17:43:13 2009] [notice] caught SIGTERM, shutting down
[Mon Sep 07 17:43:14 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations
[Mon Sep 07 17:46:13 2009] [notice] caught SIGTERM, shutting down
[Mon Sep 07 17:46:13 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations
[Tue Sep 08 11:08:00 2009] [notice] caught SIGTERM, shutting down
[Tue Sep 08 11:08:01 2009] [notice] Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch configured -- resuming normal operations

I am curious as to the significance of being able to "wake up" SSH and Apache on local connections. It seems that I am always able to access my site from my local network, since if it is down, accessing it from my local network "wakes up" the services. Is there some setting that would cause these services to go to sleep after a certain amount of time, say 5 minutes, but then wake up on local (but not remote) connections?

Wim Sturkenboom · 09-17-2009, 03:58 AM

To be honest, I'm at a loss from an apache perspective. The random restarts are very funny. And I can not think of something that will cause a wakeup.

r3sistance · 09-17-2009, 06:10 AM

When your server goes down, after it comes back up what is the uptime like? Could it be possible the entire server may be going down?

Check if their are any alternative /var/log/messages logs like perhaps messages.1 or messages.2 preferrably checking the main log file itself right after a "down event". If you are just tailing the results to 10 try doing greps for the following three things (seperately) httpd, sshd and init

dmrossi · 09-17-2009, 02:54 PM

The uptime is brief, I believe around 5 minutes. In the past I've been SSH'd into my server and after about 5 minutes the prompt just freezes up. Ctrl+C doesn't unfreeze it, and the only thing that seems to unfreeze the terminal window is quitting the terminal application and reopening it. Upon reopening and attempting to SSH back into my server, I find that the host is now unreachable. So the server has no problem letting it's services "go to sleep" even if I'm accessing them.

I'm not exactly sure what you mean by the whole server going down. During these outages, the screen on my server is blank, but that is simply because I have the screen set to go off after 10 minutes of inactivity, but other than that, it is completely responsive. Local connections are able to get through, which temporarily enable remote connections, so I don't think the entire server is going down.

I performed grep httpd, grep sshd, and grep init, all of which produced nothing. I let the commands run for a few minutes, and then canceled them after I saw nothing happen, although I'm not sure what was supposed to happen, so my apologies if I was supposed to let the commands run longer.

The other message logs contain nothing but:

Code:

Server -- MARK --

r3sistance · 09-18-2009, 08:05 AM

I meant the uptime from the command uptime

What run level are you running the server in? the screen should go blank after a while in the normal multiuser run level but I suspect your running your server in a multiuser with x run level what is less ideal from servers.

dmrossi · 09-18-2009, 11:23 AM

I hope this is correct:

Code:

who -r
         run-level 2  2009-09-16 10:34                   last=