Hey,

I have a webserver which randomly stops working, I've experienced a crash just now. I've tried restarting httpd, memcached even named, but the website does not show. Services restart fine but the site won't show. The database is offloaded to another server so haven't given a look into that. The only way I can get the site back up is to reboot the server, not touching the db server here.

Any clues as to how I could troubleshoot this?

CentOS 6.4
PHP 5.4
Plesk 11.5
ASL 4.0

Edit: I may have narrowed down the problem, based on a hunch I looked into inode usage, currently it stands at 47% in use and this right after a full reboot. How can I track down where and what is causing the Inode usage?

When you say "crash", do you mean the httpd daemon actually crashes/segfaults, and that restarting it doesn't help? Or does the web site suddenly become unresponsive with httpd still running?

Have you tried accessing the site from the server itself?

Have you tried connecting to TCP port 80 and/or port 443 (whichever is relevant for your site) with telnet (or netcat or whatever) both from a remote system and from the server itself (in the latter case trying both the public IP as well as 127.0.0.1)?

Does netstat -ltp show httpd as listening to port 80/443?

Does your server still have a working network connection when this happens? Can you reach hosts on the Internet?

Have you checked the server logs for clues?

When you say "crash", do you mean the httpd daemon actually crashes/segfaults, and that restarting it doesn't help? Or does the web site suddenly become unresponsive with httpd still running?
I suspect httpd is still running with the site being unresponsive.

Have you tried accessing the site from the server itself?
Is a remote server, but I can log in via shell

Have you tried connecting to TCP port 80 and/or port 443 (whichever is relevant for your site) with telnet (or netcat or whatever) both from a remote system and from the server itself (in the latter case trying both the public IP as well as 127.0.0.1)?
Have not tried this

Does netstat -ltp show httpd as listening to port 80/443?
Have not tried this, because of the nature of the website I cannot risk keeping the site down for troubleshooting, the client starts raising hell

Does your server still have a working network connection when this happens? Can you reach hosts on the Internet?
Is a remote server, but I can log in via shell

Have you checked the server logs for clues?
Will take another look but I took a look last time but I could not find anything in the logs

As i mentioned in the edited part of my post, I suspect the issue maybe because of the server running out of inodes. After a reboot inode consumption stands at 47%. The site is configured to use memcached, could this be causing the high inode usage? I suspect it may be php as well but I never received any of the disk full PHP errors as I've seen before in the case of php sessions not being cleared

A simple ps -ax | grep httpd should confirm that.
Then use links or lynx to access the site. It may not look pretty or be functional, but you'll quickly see whether the site can be reached or not.
A simple telnet test will tell you whether something is listening on the port or not. If it works from the server but not from the outside, it's a network issue.
I'm sure they can wait the 15 extra seconds it will take you to run the above commands before you reboot.
You should check both the apache logs and the kernel logs.

Inodes are file system resources. Each file uses an inode, and if 47% are used, your file system can hold a little more than twice the number of files currently stored on it before it runs out out. I don't quite see how memcached would cause the file system to run out of inodes, much less how a reboot would fix the issue, but then I have absolutely no experience with memcached. Could you elaborate on this?