Had a crash and need help and info, running a fsck and clearing millions of inodes
 

I have a db server its a dual xeon with 2gb ram and a Adaptec 2230SLP with 4 Fujitsu 74GB running in a raid 10 setup and a Western Digital Raptor for a backup drive.


The main os is on the raid array(everything is). I didn't make it so \var\lib\mysql was on the raid array and mount it and have the os on another drive. Now this db server has 2 large databases on it totaling in over 3 million tables and 38gb of space. The server locked up the other night and it was rebooted and so I went to run a check on it when it came up and opened up a screen session and ran "mysqlcheck -rof --all-databases" which I have done many times before nothing unusual. The database was running correctly and the forum was working fine. It usually takes about 6-8 hours to do this and about 2 hours into it I noticed the server was somewhat down but before usually I wouldn't be able to ssh in but I was able to this time. I went and checked the screen session and every table it was checking was saying that it was read only and it was going through all the tables. Going onto the forums that the databases are used for it was throwing up an error saying the sessions table was read only and couldn't write to it. Well I tried to do a shutdown -r now and reboot the server but shutdown just hung I went into top and tried to kill it and couldn't so the server was hard rebooted. It came up and said that there were errors on the drive/raid array and to run a check. Well this has been 17 hours ago that it started clearing inodes after running fsck and it has currently cleared around 6 million inodes as of 12 hours ago and counting.

Now after all this my question is do you think my databases are gone? I would hope not since the os was able to load and the server is running its just clearing inodes which means its fixing the file system. I cannot ssh in it is still running now clearing inodes.

You may well have lost data. This sounds a lot like a possible hard drive failure. I would perform some SMART tests on the individual drives to be sure.

Well its a raid 10 setup and if a hard drive failed the other array would have taken over. All the equipment is less then 2 months old. It said that the files were read only it didn't say they didn't exist or that the data is gone. Its a raid 10 array it there incase a hard drive failed and the other would take over.

can anyone else help me, it is now 1 day and 6 hours after the sever has come back online and it has cleared 6 million 19 thousand and something.