Help answer threads with 0 replies.
Go Back > Blogs > IsaacKuo
User Name


Rate this Entry

HOW-TO: Simple GRUB2 NFSroot (Debian 8 Jessie)

Posted 08-23-2016 at 03:18 PM by IsaacKuo
Updated 10-20-2016 at 10:41 AM by IsaacKuo

Sometimes you just want to boot with GRUB2 and /boot on a local drive (hard drive or USB drive), but with root on an nfs share. Other tutorials will show you how to set this up with full blown PXE network booting. This How-To shows you how to set this up with just an nfs server. Compared to a full blown PXE netboot setup:

1) Server setup is simpler. Grub nfsroot only requires an nfs server. You do NOT need a tftp server, nor do you need a customized DHCP server.

2) Some sort of local boot drive is required, because grub works with local boot devices. It may be convenient for this to be a hard drive, for large amounts of media/backup storage (performance of a local hard drive is still slow compared to an nfs share on an SSD over gigabit ethernet). Or it can be a small old USB thumbdrive - 128MB is big enough.

I find this grub nfsroot technique works really nicely with an SSD equipped laptop connected via gigabit ethernet to a client workstation. You can use the network manager GUI to easily set up "Shared with other computers" on the wired connection (IPv4 settings), and just set up an nfs share. Or if you have a gigabit switch, it's a nice way to effectively share the speed of just one SSD with several client workstations.

So, here are the steps, assuming the client and server are already set up with traditional local installs. The client install will be cloned to an nfsroot copy, but the local install will still be usable.

>>> >>> ON SERVER <<< <<<

Set up the nfs share with:

apt-get install nfs-kernel-server
mkdir /netroot
vi /etc/exports
Create the following entry in /etc/exports which will share it to 10.42.0.* (adjust according to your LAN setup)
The async option is not strictly necessary, but I think it improves performance.
systemctl restart nfs-kernel-server
>>> >>> ON CLIENT <<< <<<

I'm going to assume the OS is currently installed on sda1, with no separate /boot or /home partition. We'll first want to mount netroot.
mkdir /netroot
mkdir /mnt/sda1
vi /etc/fstab
Add /netroot to /etc/fstab with:
Code: /netroot nfs rw,noatime,nolock,noauto 1 1
Now mount netroot and copy over the OS with:
mount /netroot
rsync -vaxAX --delete --exclude tmp /. /netroot/
cp -vax /bin/* /netroot/bin/
cp -vax /usr/bin/* /netroot/usr/bin/
mkdir /netroot/tmp
The rsync command can be safely interrupted at any time, and run afterward. Unlike cp, rsync will skip files which have already been copied over completely. You can include multiple --exclude options to skip other things, such as a large media folder within the main partition.

The options in this rsync command are:

-v = verbose (so you can see the progress)
-a = archive mode (tries to preserve permissions, owner, special files, symlinks, etc)
-x = don't cross over into other partitions (this will automatically make it skip various weird partitions, as well as any large media partitions you may have set up)
-A = preserve ACLs
-X = preserve extended attributes
--delete = delete any files which no longer exist in the source
--exclude tmp = skip tmp

Later on, if you wish, you can sync up the local install to the netroot install with:
rsync -vaxAX --delete --exclude tmp --exclude etc/fstab --exclude etc/network/interfaces --exclude etc/initramfs-tools/initramfs.conf /. /netroot/
cp -vax /bin/* /netroot/bin/
cp -vax /usr/bin/* /netroot/usr/bin/
That will skip various files which we're about to modify in /netroot.

Now we are ready to make modifications to this clone to make it suitable for nfsroot.
vi /netroot/etc/fstab
Change the root entry mount point from / to /mnt/sda1. This lets you conveniently access files on the local OS partition later on. Then add the following lines:
/dev/nfs            / nfs tcp,nolock 0 0
proc /proc    proc  defaults 0 0
none /tmp     tmpfs defaults 0 0
none /var/tmp tmpfs defaults 0 0
none /media   tmpfs defaults 0 0
none /var/log tmpfs defaults 0 0
Now we need to create an entry in /netroot/etc/network/interfaces. At least for Debian 8 Jessie, this is required to prevent boot from hanging on something taking down the network interface.

First determine the name of the wired ethernet link with "ip link list" or by right clicking on network manager to see connection info. I'll assume it's eth0; adjust accordingly.
vi /netroot/etc/network/interfaces
Create or alter the eth0 entry so it looks something like this:
###allow-hotplug eth0
iface eth0 inet dhcp
Notice I have commented out "allow-hotplug eth0". If hotplug is allowed, the boot process will hang. So just don't do it.

Now, we can set up a suitable initrd for nfsroot. Even though we're not using PXE to boot, I'm naming these with PXE to remind us that the initrd and vmlinuz are identical to those used for PXE booting.
vi /netroot/etc/initramfs-tools/initramfs.conf
Add the following line at the end:
Previous versions of Debian would have a line saying BOOT=local, which you'd modify to BOOT=nfs. But not Debian 8 Jessie. Anyway, something to look out for, perhaps, if your config files are already customized, or inherited from earlier versions, or you're using this for another Debian/Ubuntu variant.

If the client has more than one network device, you may also want to modify the device line to something like:
If you don't specify this, then the client may sit around seemingly hung up during the initial DHCP and nfs root mounting phase. My experience is that it will eventually succeed in mounting the nfs root and will continue booting. But this may take several minutes of waiting! Specifying which network device to use will eliminate this waiting. You may have to use trial and error to figure out whether "eth0" or "eth1" is the correct choice here.

Whichever choice you make, this will cause some interesting effects. If you do NOT specify the device, then your network devices will be named eth0 and eth1 (and eth3... if you have even more NICs). But if you DO specify the device, then chances are only the device you specify will be named eth0/eth1. The rest will get loaded later in the boot process - after persistent naming rules are loaded. So, the other device(s) will be named something like eth5/eth6/etc...basically, it depends on whatever else may have already taken up eth0/eth1/etc slots.

mkinitramfs -d /netroot/etc/initramfs-tools -o /boot/initrd.pxe
cp -vax /boot/vmlinuz-`uname -r` /boot/vmlinuz.pxe
cp -vax /boot/*.pxe /netroot/boot/
Note that these commands copy initrd.pxe and vmlinuz.pxe from /boot to /netroot/boot/ even though they won't be utilized there. But it might be convenient to already have them there if you wish to convert this setup to a full blown PXE netboot setup someday.

The (possibly) final step is to set up the custom grub2 entry. You'll want to base the entry on an existing entry in /boot/grub/grub.cfg. The result will look something like this:
vi /etc/grub.d/40_custom
menuentry "Netroot" {
   echo 'Trying to boot via nfs ...'
   insmod gzio
   if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
   insmod part_msdos
   insmod ext2
   set root='hd0,msdos1'
   echo 'Loading Linux ...'
   linux /boot/vmlinuz.pxe root=/dev/nfs nfsroot= rw  quiet
   echo 'Loading initial ramdisk ...'
   initrd /boot/initrd.pxe
Pay attention to "set root", because this will determine where it tries to load vmlinuz.pxe and initrd.pxe from. If you have a separate /boot partition, then you'll want to change the path of /boot/vmlinuz.pxe and /boot/initrd.pxe to /vmlinuz.pxe and /initrd.pxe.

Now change the default GRUB choice with:
vi /etc/default/grub
Update /boot/grub/grub.cfg with
If you're like me, this part will take a few tries to get just right. Just boot into the local Debian install and fiddle with /etc/grub.d/40_custom until you get it right.

ADDENDUM - Updating initrd.pxe and vmlinuz.pxe

At this point, any update to the kernel, or any attempt to manually run update-grub will fail. update-grub will fail because it attempts to find the canonical path to / and it just can't. This fails regardless of whether or not /boot is on its own partition. You will either have to boot to a local hard drive install to update-grub, or you can manually update /mnt/sda1/boot/grub/grub.cfg. Or not! You really only need to copy over the vmlinuz.pxe and initrd.pxe from /boot/ to /mnt/sda1/boot/; the customized grub.cfg entry will still work.

But you still have the problem of apt-get upgrade (or apt-get dist-upgrade) complaining at you every time due to the failed attempt to run update-grub. To fix this, you simply need to move grub.cfg out of /boot/grub, with this:
mv -vi /boot/grub/grub.cfg /boot/grub/grub.cfg.MOVED
The Debian 8 post kernal install script at /etc/kernel/postinst.d/zz-update-grub checks to see whether or not /boot/grub/grub.cfg exists. If not, then it doesn't bother to try to run update-grub. Problem solved! Sort of. You still have to manually copy over vmlinuz-3.16.0-4-686-pae to vmlinuz.pxe and initrd.img-3.16.0-4-686-pae to initrd.pxe. And it can get really hairy keeping track of which version of the file has the correct settings for BOOT=nfs and so on. Basically, until you get really comfortable proceed with some caution and be prepared to boot into a local hard drive install to rescue things when something inevitably goes wrong. And make a backup folder containing a working vmlinuz.pxe initrd.pxe pair.

ADDENDUM - Random notes

I sort of messed up the naming conventions with "vmlinuz.pxe" and "initrd.pxe". In order to match the naming conventions of other contents in /boot, it should actually be "vmlinuz.pxe" and "initrd.img.pxe". I was missing the ".img". But it turns out that this is okay. If the naming conventions are correct, then "update-grub" will auto-detect the pair and will create a new grub.cfg menu entry for them. That would be nice, except the entry won't work because it lacks options telling where to find the nfs root share. So, it turns out my messed up naming works out all right in the end. You have to create your own custom boot entry anyway.
Posted in Uncategorized
Views 11414 Comments 4
« Prev     Main     Next »
Total Comments 4


  1. Old Comment
    Notes on Debian 9 Stretch

    one quick initial note on Debian 9 Stretch:

    When upgrading from Debian 8 to Debian 9, it will want to update grub-pc. It will fail, because it will try to run update-grub even if /boot/grub/grub.cfg does not exist. More generally, my trick of renaming (or erasing) /boot/grub/grub.cfg might not work any more. It seems to try to run update-grub regardless of whether or not grub.cfg exists.

    So, instead of using this trick to prevent trying and failing update-grub (fails when it can't find the root canonical path), do this instead:

    After booting up into the NFSROOT machine, uninstall grub-pc with:

    apt-get remove grub-pc
    apt-get autoremove --purge
    If you don't do this, then it's not really a problem. It's just that every time you do an update or install software with apt-get, it will complain that it can't finish installing grub-pc. If you don't mind the annoyance of this every time, you can just leave grub-pc semi-installed.
    Posted 06-22-2017 at 08:44 AM by IsaacKuo IsaacKuo is offline
  2. Old Comment
    More Debian 9 Stretch notes:

    - - - - - RESOLV.CONF PROBLEMS - - - -

    It seems that network-manager messes up name resolution with network booting. I haven't figured out an elegant solution yet, but here's what I've got...

    apt-get remove --purge grub-pc network-manager
    apt-get autoremove --purge
    After doing this, manually create /etc/resolv.conf with something like "vi /etc/resolv.conf" (or you can edit the file on the file server, with something like "vi /netroot/etc/resolv.conf")

    Make /etc/resolve.conf look something like this:
    Typically, you'll want to put the ip address of your router for the nameserver.


    Stretch no longer uses "eth0" style names. It will instead be something like "enp0s25", which you can determine by using the command:

    ip link list
    So, you'll want to create (or modify) the /etc/network/interfaces entry to look something like:

    ###allow-hotplug eth0
    iface enp0s25 inet dhcp
    - - - - - SUMMARY - - - - -

    So basically:

    1) Use enp0s25 style names for the ethernet link, rather than eth0 style name

    2) After booting up to the NETROOT install, apt-get remove grub-pc network-manager

    3) Manually create /etc/resolv.conf

    Everything else works as per the How-To
    Posted 06-27-2017 at 04:16 AM by IsaacKuo IsaacKuo is offline
    Updated 06-27-2017 at 08:35 AM by IsaacKuo
  3. Old Comment
    BTW, it may not be necessary to have the entry in /etc/network/interfaces at all. It seems to work fine without it.
    Posted 06-28-2017 at 04:37 AM by IsaacKuo IsaacKuo is offline
  4. Old Comment
    This how-to still works in Debian 10 Buster

    Also, I confirm there's no need for the entry in /etc/network/interfaces - just comment out the ethernet entry (leaving only the loopback entry). This is very conveniently portable so it's one less thing to alter when copying a setup for another computer.
    Posted 07-19-2019 at 05:19 PM by IsaacKuo IsaacKuo is offline


All times are GMT -5. The time now is 06:24 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration