Hi, I work at a small school as a Teacher and as IT. In reality there isn't much on site, and I have little experience with linux. As you will see this is a comedy of errors, but advice is very welcome!

We have a small server that is using raid 1 to mirror a disk incase of failure. This was setup by someone else before I came along, and I have only been learning the details now as I go.

Mistake 1: I added a script to startup to use 'redir' and didn't put an '&' at the end of the command, so on next startup it was just sitting there on startup before completing the boot. After some reading, I decided to use a live cd, mount the drive, and fix the script.

Mistake 2: I didnt know this server used software raid, and mounted hda1, not md0. I soon realised my error and re-fixed my script by mounting md0 from the live cd.

Mistake 3: The server is back up and running, but only on the one drive. hda is missing from /proc/mdstat . So being the clever newbie I am, I added /dev/hdc to the raid. However I did something completely wrong - somehow I made it sync with the corrupted hda, not the in use hdc.

Result: Now the server is sitting there, still working, but unable to log in as it can't access the disk.


Is the data corrupted? (I know from that previous experience I could grab the config files if I mount it as hda, etc)

And can it be repaired and put back into operation?


So I have done some reading and I beleive that I need to boot off a live cd, rebuild the superblocks on both drives, and then cross my fingers and reboot. Does this sound reasonble in theory? - or will that not be enough?


Thank you in advance for any advice,
David

For e.g. with 4 Disks in this example Disk hde1 is defect.
With raidhotremove you can remove the broken disk (but I've to shutdown the system
cause of IDE Disks and my controller cannot really hot swap.)
and then with raidhotadd Linux is issued to rebuild the raid.

$ cat /proc/mdstat

Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdk1[3] hdi1[2] hdg1[1] hde1[0](F)
234444288 blocks level 5, 32k chunk, algorithm 2 [4/3] [_UUU]
unused devices: <none>

$ raidhotremove /dev/md0 /dev/hde1
$ cat /proc/mdstat

Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdk1[3] hdi1[2] hdg1[1]
234444288 blocks level 5, 32k chunk, algorithm 2 [4/3] [_UUU]
unused devices: <none>

$ raidhotadd /dev/md0 /dev/hde1
$ cat /proc/mdstat

Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hde1[4] hdk1[3] hdi1[2] hdg1[1]
234444288 blocks level 5, 32k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 0.0% (33872/78148096) finish=76.8min speed=16936K/sec
unused devices: <none>

So has this been done? Did you check /proc/mdstat?

Thanks for your reply; I really wish I had used your instructions there first.

Its a two disk system, after stuffing the first disk by mounting it as hda rather then as md0, I then managed to add it again, but it mirrored the wrong way, ie hda->hdc, so md0 is now broken, rather then hdc->hda, as I intended. (Im learning a lot as I go here, but I messed that up completely).

(I went home with recovery on, and the next day I can't login anymore! - it just sits there when I think it should be accessing md0 until I press ctrl-c)

So now both disks are broken, and thats md0 completely out. I can mount hda1 from a live CD, and grab the configs if needed to start over, but I would really prefer repairing what I have, as I would struggle to quickly setup this box again.

Its impossible to now check /proc/mdstat as I cant login, and Im pretty sure if I restart the server it will fail on startup. It suprised me its still running fine - I guess because its not a fileserver.

Maybe you can try to login with ssh?
If ssh is enabled...

Have tried, it does the same thing; just sits there. I think it will be trying to read '.bashrc'

Does the live cd idea sound feasible to you?

In order to backup your data and then reinstall yes...
But how about if you start with the 1st OS Media CD?
I still don't know which Distribution your running?
For e.g. if it's Redhat then you should be able to boot in rescue mode, there you'll have all the tools you'll need to recreate the raid 1. So I would recommend 1st to backup with live distri. and then try to repair if that should fail then you still can reinstall but at the end you've learned something.

Ok, that sounds like a good plan.

The box is running SUSE or more specifically SLES 9.

Thanks for your help odcheck - I will try this tomorrow after school hours (and hopefully be back on my feet by the next morning).