How to recover a raid1 over nbd after failure of local disk?
 

I have a FC10 server. I have a raid1 (with bitmap) consisting of a local disk and one provided by nbd (write mostly). The raid contains a volume group, and logical volumes from this group are mounted from fstab at an early stage during the boot. The nbd component is added during init3 processing by an init.d script.

The logical volumes mount at places like /var/www/html and /var/mail.

My local disk failed (bad connection). I didn't notice for a while because the nbd mounted component continued to function OK. Then I happened to check /proc/mdstat and saw the local disk as absent.

So I tried rebooting (without having fixed the physical problem yet). The boot failed when fstab tried to mount logical volumes because the local copy wasn't working and the nbd init.d script hadn't been executed yet.

Then I fixed the physical problem and rebooted. Big mistake.

The system booted up fine, but the synchronization of the raid1 went in the wrong direction. I think this was because the local component is always added first (which was out of date). When the up-to-date component was added later, raid resolved the differences by making the new one mirror the old one.

I lost a day's worth of work.

What should I do to avoid repeating this mistake?

Thanks for any suggestions.

Adrian Stephens - Cambridge UK

Restore from backup. I think that's about all that can be said.

- Arch