I have an old HP Z600 engineering workstations I use for a multi-purpose server at home. This machine is around 12 years old but was totally kick-butt when it was new so even by todays standards its fairly powerful (dual Xeons 12 cores, 24 threads @ 3.2GHz, 48GB ECC RAM). Because of it's age though it uses a legacy BIOS for booting but I have a NVMe drive on a card in a PCI slot where Linux Mint 20.3 is installed and runs from, or did at least. It also has 4 mechanical SATA hard drives and one of them had the mbr/fat32 boot partition and grub which would then launch the kernel and desktop from the NVMe. (Can't boot directly from NVMe with legacy BIOS) This has been working for well over a year.
Alas, the hard drive with the boot partition failed. Fortunately [almost] all of my music on it was backed up. Unfortunately that little boot partition was not since I figured I could recreate it easily. And I have been trying to do just that without much success. D'oh! I created a small mbr/fat32 boot partition on the new replacement drive, made it active, set the boot flag, etc. and now it boots and just dumps me at a grub prompt, no menu. I'm also starting to wonder if I need to add an NVMe and/or btrfs driver to the boot partition since I seem unable to find or get a directory listing (ls) of the NVMe drive. Or of the ext4 partitions on the 4 hard drives either for that matter. I can access them all including the NVMe just fine from a live USB disk.
I've booted from a live USB drive and gone thru all the advanced options of boot repair experimenting with them and trying to set them to what I think are the correct settings but with no success. I've run a bunch of google searches trying to find out how to rebuild the menu and/or create an entry to launch my kernel from the NVMe but all I keep finding is articles about how to customize an existing menu and change the default. So it would seem I am just not using the appropriate search terms. Trying to run sudo update-grub from my Mint live disk results in "usr/sbin/grub-probe: error: failed to get canonical path of '/cow'. ("cow" ??)

Note: Replacement boot drive/partition = /dev/sdd1 Root partition = /dev/nvme0n1p1 Other drives/partitions are /dev/sda1, sdb1, sdc1, sdd2.

Please, oh might graybeards! Help me get my server running again! I will sacrifice a virgin if necessary.

(Debated if I should ask this in Hardware or Software but chose software because the NVMe seems to be an important aspect of the problem)

BIOS GRUB boot is a 3-step process. First the stub in the mbr loads the main GRUB kernel. That clearly is working on your system. If the process breaks there, you get the dreaded "GRUB rescue>" prompt. The second stage is for the GRUB kernel to load its shell module. A break after this gives you the "GRUB>" prompt. Finally the shell module reads the configuration file and loads the modules necessary to read filesystems, including NVMe. So where is yours breaking?

PS cow stands for copy-on-write, a facility that some filesystems have.

1 members found this post helpful.

You lost /boot which probably contained grub and also the kernel. I suggest that you use your live USB system to mount each of your partitions one by one and list the directories you have in each partition. In particular you should be looking for grub, linux, initrd, etc to see if you have them and if so where are they? Looking for these things through the grub prompt is maddening.

1 members found this post helpful.

You can use a live CD and then chroot to your NVME drive and try reinstalling grub on one of the other drives. This link has a nice how-to: https://www.bleepingcomputer.com/for...hroot-command/

1 members found this post helpful.

I would have done a file check 1st before messing with mbr myself. I have broke boot many times with a corrupted file system from improper shutdown. Especially on ext2 partitions.

Assuming the boot partition(sdd1) is mounted to /boot, it can be mounted to /mnt/boot before chrooting or /boot after chrooting.

1 members found this post helpful.

Copy on Write. What that is telling you is that you are trying to install grub on the 'live' medium you are using which will never work as it is read only. You need to chroot to the system boot partition which is explained at the link below.

https://help.ubuntu.com/community/Grub2/Installing

Since you have boot repair, is there any reason you didn't run the Create BootInfo Summary and post a link to the output here for members to review?

1 members found this post helpful.

First of all, thanks to everyone for their replies!
Hazel: I suspected as much, but thanks for the extra details and explanation.
Jailbait: So of course I lost /boot when that hard drive failed. Checking the NVMe drive from the live disk I find just a @ and a @home folders. Under @ I'm finding most of the things I would expect including bin, boot, dev etc, home, lib and lib32 and lib64, media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, and var. Does not contain grub, linux, or initrd. So I believe you are correct that the kernel was also on the failed boot hard drive. And I suspect it may need to be in order to recognize the NVMe drive, and the Ext4 and Btrfs partitions on the other hard drives. My goal though is to have the bare minimum on the boot mechanical hard disk and once the basic booting is done run everything from the NVMe which has literally 50+ times the [sustained] transfer rate.

What a long and frustrating day. Various family mini-emergencies (to them) kept breaking my concentration. I started with the article linked by Yancek which, while much longer and more complicated, had more "in case of this..." scenarios. After some struggles to figure out where everything was and get it mounted with correct paths in the end if failed with a pair of errors regarding not understanding btrfs or fat. I saved everything I did and the associated responses to a text file on one of the other (non-boot and non-root) drives so I could post it here. And then when I went to retrieve it, no trace of it despite no indications that it failed to save.

Next I moved on to the article linked by Utek. Again I had to fiddle with paths a bit but after the previous run I knew where everything was. When I attempted to run the response was which I'm guessing is because when I mounted the NVMe as the first step in the instructions it mounted it to /mnt/@ apparently because it is formatted btrfs (which I am now regretting). Again I attempted to save my work and again it disappeared without a trace.

I am now going to have a cocktail and watch some Simpsons. Tomorrow I shall implement Plan C which is to create another partition on the NVMe (I have plenty of space) and reinstall Mint from scratch using the "Something Else" option, just like I did it originally, putting /boot on /dev/sdd1 and /root on the new /dev/nvme0n1p3 (and /swap on /dev/nvme0n1p2). Once the install is done I will attempt to modify grub menu to launch the old nvme0n1p1 installation instead of the new nvme0n1p3. I think/hope that is within my limited skill set. I am sort of expecting the grub menu to show both installations. Anyone see a problem with this plan?

Yansec: Alas, the ethernet port on the motherboard is no good so I have an HP NC365T 4 port server class network card installed. Wanted the extra ports for possible future routing and/or network segmentation abilities. And alas, the live disk will only recognize the [bad] port on the motherboard so I have no network/internet while using the live boot disk. I suppose I could save it locally, copy it to flash drive, and post it from one of my other machines...

You may also have to re-install the kernels and edit the /etc/fstab for different uuids for the /boot partition on the old mint

You will need to mount the btrfs subvolume labeled @ onto /mnt. Then mount the one labeled @home to /mnt/home. then your new /boot (sdd1) to /mnt/boot. That should get your file system structure properly mounted under /mnt.

Once that is done then you need to create the runtime environment by bind mounting /dev, /proc, /sys, and probably /run onto the appropriate locations in /mnt. That gets you to the point where you can do a chroot and be able to work within the installed system to recreate all the missing data from /boot.

The key here is that until the kernel and appropriate drivers are loaded the system cannot see the nvme drive so grub and the kernel (initfs and vmlinuz) must reside on the spinning disk (/boot) that bios can access.

1 members found this post helpful.

Woo-Hoo! Finally after 2 weeks of cursing my server is back online!
"Plan C" worked and it was relatively quick and easy. Wish I'd just done that to begin with. Oh well, I learned some stuff. Alas, my old mint green colored flash drive with my Mint live boot finally gave up the ghost from all this rebooting. RIP old friend, you led a long and productive life.

Once again, Thank You everyone!