Originally Posted by mostlyharmless
Probably this is a simple minded question, but if you DO add them sequentially, and let them resync fully, then shutdown and restart the system, does mdadm still refuse to add them?
BTW, I'm guessing the only reason they'd get out of sync is that /dev/nbd0 would miss writes if the network goes down, so how would it get more up to date?
To answer the last question: nbd0 can be more up to date if there is a failure of the local disk. I think I've got an unreliable SATA connector, but I've certainly seen failures of the local hard disk component. When this happens nbd0 continues to support the raid component (as it's supposed to), and then the local disk is out of date.
Nothing I found works automatically. I can force the correct update by adding nbd0 first with --run, and then sdb6 with --re-add.
I think the moral of this story is: 1. make sure the raid is started after init 2 processing. After a RAID failure, to boot into single mode (init 2 I think), use mdadm --examine to determine which is the more up to date component, manually assemble --run the more up to date component, then --re-add the out of date one. Wait for the sync and reboot into normal operation.
By the way, I had a hang about once per day. Although I've not localized it, I suspect the raid/nbd combination, and I'm going to remove the nbd raid component, and instead do a periodic unison backup to the remote server. Less high tech, but I'm guessing I'll have less trouble with it.