HOW-TO: Simple GRUB2 NFSroot (Debian 8 Jessie)
Sometimes you just want to boot with GRUB2 and /boot on a local drive (hard drive or USB drive), but with root on an nfs share. Other tutorials will show you how to set this up with full blown PXE network booting. This How-To shows you how to set this up with just an nfs server. Compared to a full blown PXE netboot setup:
1) Server setup is simpler. Grub nfsroot only requires an nfs server. You do NOT need a tftp server, nor do you need a customized DHCP server.
2) Some sort of local boot drive is required, because grub works with local boot devices. It may be convenient for this to be a hard drive, for large amounts of media/backup storage (performance of a local hard drive is still slow compared to an nfs share on an SSD over gigabit ethernet). Or it can be a small old USB thumbdrive - 128MB is big enough.
I find this grub nfsroot technique works really nicely with an SSD equipped laptop connected via gigabit ethernet to a client workstation. You can use the network manager GUI to easily set up "Shared with other computers" on the wired connection (IPv4 settings), and just set up an nfs share. Or if you have a gigabit switch, it's a nice way to effectively share the speed of just one SSD with several client workstations.
So, here are the steps, assuming the client and server are already set up with traditional local installs. The client install will be cloned to an nfsroot copy, but the local install will still be usable.
>>> >>> ON SERVER <<< <<<
Set up the nfs share with:
Create the following entry in /etc/exports which will share it to 10.42.0.* (adjust according to your LAN setup)
The async option is not strictly necessary, but I think it improves performance.
>>> >>> ON CLIENT <<< <<<
I'm going to assume the OS is currently installed on sda1, with no separate /boot or /home partition. We'll first want to mount netroot.
Add /netroot to /etc/fstab with:
Now mount netroot and copy over the OS with:
The rsync command can be safely interrupted at any time, and run afterward. Unlike cp, rsync will skip files which have already been copied over completely. You can include multiple --exclude options to skip other things, such as a large media folder within the main partition.
The options in this rsync command are:
-v = verbose (so you can see the progress)
-a = archive mode (tries to preserve permissions, owner, special files, symlinks, etc)
-x = don't cross over into other partitions (this will automatically make it skip various weird partitions, as well as any large media partitions you may have set up)
-A = preserve ACLs
-X = preserve extended attributes
--delete = delete any files which no longer exist in the source
--exclude tmp = skip tmp
Later on, if you wish, you can sync up the local install to the netroot install with:
That will skip various files which we're about to modify in /netroot.
Now we are ready to make modifications to this clone to make it suitable for nfsroot.
Change the root entry mount point from / to /mnt/sda1. This lets you conveniently access files on the local OS partition later on. Then add the following lines:
Now we need to create an entry in /netroot/etc/network/interfaces. At least for Debian 8 Jessie, this is required to prevent boot from hanging on something taking down the network interface.
First determine the name of the wired ethernet link with "ip link list" or by right clicking on network manager to see connection info. I'll assume it's eth0; adjust accordingly.
Create or alter the eth0 entry so it looks something like this:
Notice I have commented out "allow-hotplug eth0". If hotplug is allowed, the boot process will hang. So just don't do it.
Now, we can set up a suitable initrd for nfsroot. Even though we're not using PXE to boot, I'm naming these with PXE to remind us that the initrd and vmlinuz are identical to those used for PXE booting.
Add the following line at the end:
Previous versions of Debian would have a line saying BOOT=local, which you'd modify to BOOT=nfs. But not Debian 8 Jessie. Anyway, something to look out for, perhaps, if your config files are already customized, or inherited from earlier versions, or you're using this for another Debian/Ubuntu variant.
If the client has more than one network device, you may also want to modify the device line to something like:
If you don't specify this, then the client may sit around seemingly hung up during the initial DHCP and nfs root mounting phase. My experience is that it will eventually succeed in mounting the nfs root and will continue booting. But this may take several minutes of waiting! Specifying which network device to use will eliminate this waiting. You may have to use trial and error to figure out whether "eth0" or "eth1" is the correct choice here.
Whichever choice you make, this will cause some interesting effects. If you do NOT specify the device, then your network devices will be named eth0 and eth1 (and eth3... if you have even more NICs). But if you DO specify the device, then chances are only the device you specify will be named eth0/eth1. The rest will get loaded later in the boot process - after persistent naming rules are loaded. So, the other device(s) will be named something like eth5/eth6/etc...basically, it depends on whatever else may have already taken up eth0/eth1/etc slots.
Note that these commands copy initrd.pxe and vmlinuz.pxe from /boot to /netroot/boot/ even though they won't be utilized there. But it might be convenient to already have them there if you wish to convert this setup to a full blown PXE netboot setup someday.
The (possibly) final step is to set up the custom grub2 entry. You'll want to base the entry on an existing entry in /boot/grub/grub.cfg. The result will look something like this:
Pay attention to "set root", because this will determine where it tries to load vmlinuz.pxe and initrd.pxe from. If you have a separate /boot partition, then you'll want to change the path of /boot/vmlinuz.pxe and /boot/initrd.pxe to /vmlinuz.pxe and /initrd.pxe.
Now change the default GRUB choice with:
Update /boot/grub/grub.cfg with
If you're like me, this part will take a few tries to get just right. Just boot into the local Debian install and fiddle with /etc/grub.d/40_custom until you get it right.
ADDENDUM - Updating initrd.pxe and vmlinuz.pxe
At this point, any update to the kernel, or any attempt to manually run update-grub will fail. update-grub will fail because it attempts to find the canonical path to / and it just can't. This fails regardless of whether or not /boot is on its own partition. You will either have to boot to a local hard drive install to update-grub, or you can manually update /mnt/sda1/boot/grub/grub.cfg. Or not! You really only need to copy over the vmlinuz.pxe and initrd.pxe from /boot/ to /mnt/sda1/boot/; the customized grub.cfg entry will still work.
But you still have the problem of apt-get upgrade (or apt-get dist-upgrade) complaining at you every time due to the failed attempt to run update-grub. To fix this, you simply need to move grub.cfg out of /boot/grub, with this:
The Debian 8 post kernal install script at /etc/kernel/postinst.d/zz-update-grub checks to see whether or not /boot/grub/grub.cfg exists. If not, then it doesn't bother to try to run update-grub. Problem solved! Sort of. You still have to manually copy over vmlinuz-3.16.0-4-686-pae to vmlinuz.pxe and initrd.img-3.16.0-4-686-pae to initrd.pxe. And it can get really hairy keeping track of which version of the file has the correct settings for BOOT=nfs and so on. Basically, until you get really comfortable proceed with some caution and be prepared to boot into a local hard drive install to rescue things when something inevitably goes wrong. And make a backup folder containing a working vmlinuz.pxe initrd.pxe pair.
ADDENDUM - Random notes
I sort of messed up the naming conventions with "vmlinuz.pxe" and "initrd.pxe". In order to match the naming conventions of other contents in /boot, it should actually be "vmlinuz.pxe" and "initrd.img.pxe". I was missing the ".img". But it turns out that this is okay. If the naming conventions are correct, then "update-grub" will auto-detect the pair and will create a new grub.cfg menu entry for them. That would be nice, except the entry won't work because it lacks options telling where to find the nfs root share. So, it turns out my messed up naming works out all right in the end. You have to create your own custom boot entry anyway.
1) Server setup is simpler. Grub nfsroot only requires an nfs server. You do NOT need a tftp server, nor do you need a customized DHCP server.
2) Some sort of local boot drive is required, because grub works with local boot devices. It may be convenient for this to be a hard drive, for large amounts of media/backup storage (performance of a local hard drive is still slow compared to an nfs share on an SSD over gigabit ethernet). Or it can be a small old USB thumbdrive - 128MB is big enough.
I find this grub nfsroot technique works really nicely with an SSD equipped laptop connected via gigabit ethernet to a client workstation. You can use the network manager GUI to easily set up "Shared with other computers" on the wired connection (IPv4 settings), and just set up an nfs share. Or if you have a gigabit switch, it's a nice way to effectively share the speed of just one SSD with several client workstations.
So, here are the steps, assuming the client and server are already set up with traditional local installs. The client install will be cloned to an nfsroot copy, but the local install will still be usable.
>>> >>> ON SERVER <<< <<<
Set up the nfs share with:
Code:
apt-get install nfs-kernel-server mkdir /netroot vi /etc/exports
Code:
/netroot/ 10.42.0.0/255.255.255.0(rw,async,no_root_squash,no_subtree_check)
Code:
systemctl restart nfs-kernel-server
I'm going to assume the OS is currently installed on sda1, with no separate /boot or /home partition. We'll first want to mount netroot.
Code:
mkdir /netroot mkdir /mnt/sda1 vi /etc/fstab
Code:
10.42.0.1:/netroot /netroot nfs rw,noatime,nolock,noauto 1 1
Code:
mount /netroot rsync -vaxAX --delete --exclude tmp /. /netroot/ cp -vax /bin/* /netroot/bin/ cp -vax /usr/bin/* /netroot/usr/bin/ mkdir /netroot/tmp
The options in this rsync command are:
-v = verbose (so you can see the progress)
-a = archive mode (tries to preserve permissions, owner, special files, symlinks, etc)
-x = don't cross over into other partitions (this will automatically make it skip various weird partitions, as well as any large media partitions you may have set up)
-A = preserve ACLs
-X = preserve extended attributes
--delete = delete any files which no longer exist in the source
--exclude tmp = skip tmp
Later on, if you wish, you can sync up the local install to the netroot install with:
Code:
rsync -vaxAX --delete --exclude tmp --exclude etc/fstab --exclude etc/network/interfaces --exclude etc/initramfs-tools/initramfs.conf /. /netroot/ cp -vax /bin/* /netroot/bin/ cp -vax /usr/bin/* /netroot/usr/bin/
Now we are ready to make modifications to this clone to make it suitable for nfsroot.
Code:
vi /netroot/etc/fstab
Code:
/dev/nfs / nfs tcp,nolock 0 0 proc /proc proc defaults 0 0 none /tmp tmpfs defaults 0 0 none /var/tmp tmpfs defaults 0 0 none /media tmpfs defaults 0 0 none /var/log tmpfs defaults 0 0
First determine the name of the wired ethernet link with "ip link list" or by right clicking on network manager to see connection info. I'll assume it's eth0; adjust accordingly.
Code:
vi /netroot/etc/network/interfaces
Code:
###allow-hotplug eth0 iface eth0 inet dhcp
Now, we can set up a suitable initrd for nfsroot. Even though we're not using PXE to boot, I'm naming these with PXE to remind us that the initrd and vmlinuz are identical to those used for PXE booting.
Code:
vi /netroot/etc/initramfs-tools/initramfs.conf
Code:
BOOT=nfs
If the client has more than one network device, you may also want to modify the device line to something like:
Code:
DEVICE=eth1
Whichever choice you make, this will cause some interesting effects. If you do NOT specify the device, then your network devices will be named eth0 and eth1 (and eth3... if you have even more NICs). But if you DO specify the device, then chances are only the device you specify will be named eth0/eth1. The rest will get loaded later in the boot process - after persistent naming rules are loaded. So, the other device(s) will be named something like eth5/eth6/etc...basically, it depends on whatever else may have already taken up eth0/eth1/etc slots.
Code:
mkinitramfs -d /netroot/etc/initramfs-tools -o /boot/initrd.pxe cp -vax /boot/vmlinuz-`uname -r` /boot/vmlinuz.pxe cp -vax /boot/*.pxe /netroot/boot/
The (possibly) final step is to set up the custom grub2 entry. You'll want to base the entry on an existing entry in /boot/grub/grub.cfg. The result will look something like this:
Code:
vi /etc/grub.d/40_custom
Code:
menuentry "Netroot" {
echo 'Trying to boot via nfs ...'
load_video
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
echo 'Loading Linux ...'
linux /boot/vmlinuz.pxe root=/dev/nfs nfsroot=10.42.0.1:/netroot rw quiet
echo 'Loading initial ramdisk ...'
initrd /boot/initrd.pxe
}
Now change the default GRUB choice with:
Code:
vi /etc/default/grub
Code:
GRUB_DEFAULT="Netroot"
Code:
update-grub
ADDENDUM - Updating initrd.pxe and vmlinuz.pxe
At this point, any update to the kernel, or any attempt to manually run update-grub will fail. update-grub will fail because it attempts to find the canonical path to / and it just can't. This fails regardless of whether or not /boot is on its own partition. You will either have to boot to a local hard drive install to update-grub, or you can manually update /mnt/sda1/boot/grub/grub.cfg. Or not! You really only need to copy over the vmlinuz.pxe and initrd.pxe from /boot/ to /mnt/sda1/boot/; the customized grub.cfg entry will still work.
But you still have the problem of apt-get upgrade (or apt-get dist-upgrade) complaining at you every time due to the failed attempt to run update-grub. To fix this, you simply need to move grub.cfg out of /boot/grub, with this:
Code:
mv -vi /boot/grub/grub.cfg /boot/grub/grub.cfg.MOVED
ADDENDUM - Random notes
I sort of messed up the naming conventions with "vmlinuz.pxe" and "initrd.pxe". In order to match the naming conventions of other contents in /boot, it should actually be "vmlinuz.pxe" and "initrd.img.pxe". I was missing the ".img". But it turns out that this is okay. If the naming conventions are correct, then "update-grub" will auto-detect the pair and will create a new grub.cfg menu entry for them. That would be nice, except the entry won't work because it lacks options telling where to find the nfs root share. So, it turns out my messed up naming works out all right in the end. You have to create your own custom boot entry anyway.
Total Comments 4
Comments
-
Notes on Debian 9 Stretch
one quick initial note on Debian 9 Stretch:
When upgrading from Debian 8 to Debian 9, it will want to update grub-pc. It will fail, because it will try to run update-grub even if /boot/grub/grub.cfg does not exist. More generally, my trick of renaming (or erasing) /boot/grub/grub.cfg might not work any more. It seems to try to run update-grub regardless of whether or not grub.cfg exists.
So, instead of using this trick to prevent trying and failing update-grub (fails when it can't find the root canonical path), do this instead:
After booting up into the NFSROOT machine, uninstall grub-pc with:
If you don't do this, then it's not really a problem. It's just that every time you do an update or install software with apt-get, it will complain that it can't finish installing grub-pc. If you don't mind the annoyance of this every time, you can just leave grub-pc semi-installed.Code:apt-get remove grub-pc apt-get autoremove --purge
Posted 06-22-2017 at 08:44 AM by IsaacKuo 
-
More Debian 9 Stretch notes:
- - - - - RESOLV.CONF PROBLEMS - - - -
It seems that network-manager messes up name resolution with network booting. I haven't figured out an elegant solution yet, but here's what I've got...
After doing this, manually create /etc/resolv.conf with something like "vi /etc/resolv.conf" (or you can edit the file on the file server, with something like "vi /netroot/etc/resolv.conf")Code:apt-get remove --purge grub-pc network-manager apt-get autoremove --purge
Make /etc/resolve.conf look something like this:
Typically, you'll want to put the ip address of your router for the nameserver.Code:nameserver 10.42.0.1
- - - - - NEW ETHERNET DEVICE NAMING CONVENTION - - - -
Stretch no longer uses "eth0" style names. It will instead be something like "enp0s25", which you can determine by using the command:
So, you'll want to create (or modify) the /etc/network/interfaces entry to look something like:Code:ip link list
- - - - - SUMMARY - - - - -Code:###allow-hotplug eth0 iface enp0s25 inet dhcp
So basically:
1) Use enp0s25 style names for the ethernet link, rather than eth0 style name
2) After booting up to the NETROOT install, apt-get remove grub-pc network-manager
3) Manually create /etc/resolv.conf
Everything else works as per the How-ToPosted 06-27-2017 at 04:16 AM by IsaacKuo
Updated 06-27-2017 at 08:35 AM by IsaacKuo -
BTW, it may not be necessary to have the entry in /etc/network/interfaces at all. It seems to work fine without it.
Posted 06-28-2017 at 04:37 AM by IsaacKuo
-
This how-to still works in Debian 10 Buster
Also, I confirm there's no need for the entry in /etc/network/interfaces - just comment out the ethernet entry (leaving only the loopback entry). This is very conveniently portable so it's one less thing to alter when copying a setup for another computer.Posted 07-19-2019 at 05:19 PM by IsaacKuo
