Hello,

Our dedicated server (that we hire from GoDaddy) crashes every few days, and needs a power cycle, when this happens we are unsure what happens, but HTTP, MySQL, FTP, SSH all shutdown.

We've never been on the site when this happens (it usually crashes when the Europeans sleep). That is, until today, I was there to witness the errors.

That is, I saw the error given by MySQL when PHP's query failed:
File './database/table.MYD' not found (Errcode: 30)

(substituted the real names for database and table)



Also was PHP's error, also saying Error 30.

Warning: Unknown: open(/var/lib/php/session/sess_9hge1cjlfbnc5d8n665afnh4t2, O_RDWR) failed: Read-only file system (30) in Unknown on line 0

Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0



It then seemed to be apparent that the filesystem had gone into read-only mode.

At the next refresh, the server had gone down again. Power cycle as normal.

I checked dmesg closely after the reboot (by doing dmesg | less so I could see it paginated).

I'll give you the dmesg at pastebin: http://pastebin.com/3VpVP095

Here's what caught my eye:
EXT4-fs (sda3): INFO: recovery required on readonly filesystem
EXT4-fs (sda3): write access will be enabled during recovery
EXT4-fs (sda3): orphan cleanup on readonly fs
EXT4-fs (sda3): ext4_orphan_cleanup: deleting unreferenced inode 27133185
EXT4-fs (sda3): ext4_orphan_cleanup: deleting unreferenced inode 27132726
EXT4-fs (sda3): ext4_orphan_cleanup: deleting unreferenced inode 27131921
EXT4-fs (sda3): ext4_orphan_cleanup: deleting unreferenced inode 27131920
EXT4-fs (sda3): ext4_orphan_cleanup: deleting unreferenced inode 27131914
EXT4-fs (sda3): 5 orphan inodes deleted
EXT4-fs (sda3): recovery complete
EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts:


That's all we know, sadly. Does anyone have any idea why this might be happening? If you need more information ask for it, I'm not sure what I should be attaching.


Cheers,
Oliver

Definitely start looking at the logs eg /var/log/messages for a start.
Also the Apache error_log and mysqld.log.
Could easily be the disk(s) filling up from a runaway process.

Next time, wait until it crashes, then DON'T reboot. Instead, boot from a rescue disk or use Rescue mode from the install media and check logs, disk space etc from there. That way you get to see what it looked like when it crashed; a normal reboot cleans up the system somewhat, so making it harder to debug.

There are many monitoring tools you could add after that, but its easier if you at least have some idea what to monitor, so you don't get flooded with info.

Do make sure you backups are up to date right now(!); sooner or later a crash is going to lose something important.