Apache just dies sometimes

fruitwerks · 06-23-2011, 01:01 PM

I have been looking into this issue for a long time now and I still can't get to the bottom of it.

The server is Ubuntu 10.04 apache is 2.2 branch...

When it happens, load is not high and memory / swap has plenty of room. Also apache will not deny the connection, it will give the browser some sign of hope because the page will not load or timeout. Nothing will load, not even the apache server-status page.

The second I restart apache, the browser gives the error like the site is down.

Any suggestions of what to look at?

Thanks!

TB0ne · 06-23-2011, 03:24 PM

Quote:

Originally Posted by fruitwerks

I have been looking into this issue for a long time now and I still can't get to the bottom of it.
The server is Ubuntu 10.04 apache is 2.2 branch...

When it happens, load is not high and memory / swap has plenty of room. Also apache will not deny the connection, it will give the browser some sign of hope because the page will not load or timeout. Nothing will load, not even the apache server-status page.

The second I restart apache, the browser gives the error like the site is down.
Any suggestions of what to look at?
Thanks!

The Apache logs would be the very first thing I'd check...but it sounds like you've checked the obvious, and have ruled out the network, system load, memory, and swap. How about disk space, just on an off chance?? Also, depending on how often it happens, have you considered starting Apache with more verbosity in the logs, to see if you catch anything different?
http://www.devshed.com/c/a/Apache/Logging-in-Apache/

Also, some of this may depend on what the page(s) are serving up. Are they PHP, Java, Javascript, and/or database dependent?? Any one of those could be a factor. Restarting Apache may just kill the bad process that's locking things up...

fruitwerks · 06-23-2011, 06:03 PM

Yes, I forgot to mention how much I have put into this. I have sat and tailed the verbose logs - after three times of seeing anything before it happens - I gave up on that front.

Disk space is plentiful, swap / ram is always good - not using selinux or apparmour, ulimit is never reached. No quotas or anything else going on. It is not a network issue - if it is, it is the strangest one I have ever seen. Using a hardware firewall that has never been suspected of anything odd. From the shell (local) I can't load a page using telnet or links / lynx. Those behave similar to a modern browser as I originally explained. Not using any php caches either.

Most of the pages are php and cgi, no java, some mysql, some postgres connections, but these continue to work when apache fails.

If anyone has some more ideas or even some insight

Valery Reznic · 06-24-2011, 12:09 AM

Quote:

Originally Posted by fruitwerks

Yes, I forgot to mention how much I have put into this. I have sat and tailed the verbose logs - after three times of seeing anything before it happens - I gave up on that front.

Disk space is plentiful, swap / ram is always good - not using selinux or apparmour, ulimit is never reached. No quotas or anything else going on. It is not a network issue - if it is, it is the strangest one I have ever seen. Using a hardware firewall that has never been suspected of anything odd. From the shell (local) I can't load a page using telnet or links / lynx. Those behave similar to a modern browser as I originally explained. Not using any php caches either.

Most of the pages are php and cgi, no java, some mysql, some postgres connections, but these continue to work when apache fails.

If anyone has some more ideas or even some insight

Do you use PHP as module or as CGI?

fruitwerks · 06-24-2011, 06:29 AM

Quote:

Do you use PHP as module or as CGI?

With that, I will mention that this server is running Plesk. I have had a few Plesk developers look into this and they all agree it has nothing to do with Plesk. Most sites are running as a module, but some may be running as CGI / FCGI.

I will also note that we also run a lighttpd instance on the same machine, bound to a different IP. This site relies heavily on PHP and it continues to function while apache is down - and this is unrelated to lighttpd because this issue was present before we started running lighttpd.

Thanks

Valery Reznic · 06-24-2011, 07:45 AM

Quote:

Originally Posted by fruitwerks

With that, I will mention that this server is running Plesk. I have had a few Plesk developers look into this and they all agree it has nothing to do with Plesk. Most sites are running as a module, but some may be running as CGI / FCGI.

I will also note that we also run a lighttpd instance on the same machine, bound to a different IP. This site relies heavily on PHP and it continues to function while apache is down - and this is unrelated to lighttpd because this issue was present before we started running lighttpd.

Thanks

I think I miss something - if Apache is down how anything website-related can function?

fruitwerks · 06-24-2011, 08:06 AM

I was just noting that if it is somehow a PHP related issue, that the lighttpd instance is not affected by it, it is a separate server on the same machine.

Valery Reznic · 06-24-2011, 08:17 AM

Quote:

Originally Posted by fruitwerks

I was just noting that if it is somehow a PHP related issue, that the lighttpd instance is not affected by it, it is a separate server on the same machine.

Apache used by default on Ubuntu is multi-threade (it uses MPM worker) PHP is known to be not thread safe.
You can try to use apache with MPM-prefork instead and see if it help.

fruitwerks · 06-24-2011, 08:24 AM

Yes, we are using prefork. here is a modules list.

Code:

 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 bw_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 fcgid_module (shared)
 include_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 perl_module (shared)
 php5_module (shared)
 python_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 status_module (shared)
 suexec_module (shared)
 userdir_module (shared)
 wsgi_module (shared)

If anyone thinks some of these may not be required, I will double check - but Plesk is needy so this is likely fairly minimal