dracut disk /dev/disk/by-uuid/blah missing when its not
 

I have a server setup with 2 1TB Drives using software raid1. 3 partitions total...

I reboot my server after a couple months.. and now it is failing to boot. It drops into a dracut emergency mode shell.

The error indicates that the disk /dev/disk/by-uuid/8046c24a-cf00-4c82-bb34-f8b76cb29 does not exist. However when I boot into the rescue kernel and do blkid I can see that it does exist as the root partition device /dev/md127.

I would paste more details from commands but I can't copy and paste from the Java console application that is the only way I can access the server right now. I'll happily type out anything that would be helpful.. and I can screenshot easy enough as well.

I tried rebuilding the initramfs image with dracut as suggested in several posts I found but I am still getting the same error. I am out of ideas on what to do at this point and welcome any guidance.

The only thing I see that seems odd is when I look at the /run/initramfs/rdsosreport.txt at the very beginning it does ls -l /dev/disk/by* and it gets back a no such file or directory error. This seems like it could cause the disk not being found but I don't know why it would not find it.. when i do the same command in the rescue kernel i get a full list which has all the devices and raid devices.

The section of the log goes like this:

I have spent more hours on this then I care to think about so if it sounds like i'm frustrated.. I am. I am hoping I'm just missing something.. obvious.

Sounds like a race - are the devices externally connected ?. The devices nodes may be taking a while to get created.

They are not externally connected. They are plugged into the hard drive slots of a C6100 non-removable Blade chassis. And sadly i'm losing the race.. every time.

If they are taking a while to get created, is there something I can add to the boot config to create a delay? allowing the nodes to be created? I realize this is more of a workaround but... Id really like to get the server up at least.

Maybe try adding "rootdelay=15" (that's seconds) to the kernel parms. Dunno, but easy to try.

So I can only get into the rescue shell.. and it looks like it doesn't support ext4 so I can't mount the boot partition... in the rescue shell i have a /boot with images but no grub config file..

Suggestions on how I can go about adding kernel parameters?

I am trying to boot from installation media now to see if that will let me access the grub config file now.

Edit: I was able to add a new menu entry including the delay, however that new menu entry did not show up after I ran the grub2-mkconfig command to generate the new grub.cfg file. I verified it was in the .cfg file but it didn't show up when I booted. So I used the grub option to edit an existing entry and added the rootdelay=15 to it... however that doesn't seem to have made any difference. I still see dracut-initqueue timeout messages during boot and end up in the same dracut shell with the same message indicating the disk doesnt exist. I still see the same cannot access /dev/disk/by*: No such file or directory message in the rdsosreport.txt file as well.

I upped the rootdelay to 90 but it doesnt seem like its making a difference in how long the system takes to boot or the results I am getting. :(

In case it wasn't clear this is a software raid setup. The supposed missing disk is the root partition in the array. All disks for the array show up in blkid and the drives/volumes show clean and intact according to mdraid. I am still stumped on this and welcome any other ideas or requests for information.

In case anyone else encounters this... i solved the issue by changing the IDE mode in the BIOS from AHCI to IDE. Turns out my provider moved my server to a different but identical set of hardware... which had the AHCI option set in BIOS.