NFS-RAMBOOT - How-To for Debian 9 Stretch

Posted 06-30-2017 at 05:09 PM by IsaacKuo

Tags diskless, nfsroot, pxe, ramboot, tmpfs

NFS-RAMBOOT adapts my RAMBOOT hack to NFS root (PXE or not). On boot, it extracts an OS tarball from an nfs share into local RAM (recommended 3+GB of RAM). The result is like a very fast SSD.

Why do this? Compared to RAMBOOT, loading (and saving) the OS tarball over gigabit ethernet can be faster than a local hard drive or USB drive, if the file server has an SSD or a faster hard drive (compare 3.5" drives to 2.5" laptop drives, for example). Also, having the OS tarballs on an nfs server can be more convenient for maintenance, archiving, etc.

This how-to will assume the use of local /boot with GRUB. For setting up PXE, see my blog about PXE boot setup.

To recap (for those unfamiliar with RAMBOOT), running from a tmpfs root in RAM is like an extremely fast SSD, but with some important differences:

1) Because tmpfs is purely in RAM, you lose all data on power down. A script can save the current / partition into an updated ramboot/image.tar.gz, but this takes time.

2) Boot time is slower due to the need to load the entire image.tar.gz file over the network. A normal SSD just mounts and immediately continues with the boot process.

3) Because the OS partition resides entirely in RAM, less RAM is available for normal use. However, this can be compensated for by using a swapfile or partition. This swapfile may be on another computer, and may even be put on the RAM of another computer (see my other blog entries for how to set this up).

4) Because / is not a "normal" partition, update-grub would fail when it tries to figure out the canonical path of /. To compensate, we'll apt-get remove grub-pc. You'll have to manually copy over updated initrd and vmlinuz files to the physical boot drive.

- - - - - 8< - - - - - cute here - - - - - 8< - - - - -

Step 0) SET UP NFS SERVER

On the server, set up the nfs share with:

Code:

apt-get install nfs-kernel-server
mkdir /netroot
mkdir /netroot/ramboot
vi /etc/exports

Create the following entry in /etc/exports which will share it to 10.42.0.* (adjust according to your LAN setup)

Code:

/netroot/ 10.42.0.0/255.255.255.0(rw,async,no_root_squash,no_subtree_check)

The async option is not strictly necessary, but I think it improves performance.

Code:

systemctl restart nfs-kernel-server

That's it for the file server!

Step 1) MINIMAL DEBIAN 9 INSTALL

On client, do a minimal Debian 9 install on /dev/sda1 (either a hard drive or USB, typically) and boot to it. Deselect all software suites, unless you're sure you want them.

Step 2) PREPARE /etc/fstab

Log in as root. Edit /etc/fstab and add these lines: (adjust for server IP address)

Code:

#none /       tmpfs size=95% 0 1
none /tmp     tmpfs size=95% 0 1
none /var/tmp tmpfs size=95% 0 1
none /var/log tmpfs size=95% 0 1
10.42.0.1:/netroot /netroot nfs rw,noatime,nolock,noauto 1 1

The resulting fstab will look something like:

Code:

UUID=5129efa8-fd32-58e4-83fd-2e3afdbcdef653 /          ext4 error=remount-ro 0 1
#none /       tmpfs size=95% 0 1
none /tmp     tmpfs size=95% 0 1
none /var/tmp tmpfs size=95% 0 1
none /var/log tmpfs size=95% 0 1
10.42.0.1:/netroot /netroot nfs rw,noatime,nolock,noauto 1 1

Make a couple copies with:

Code:

cp /etc/fstab /etc/fstab.NORMAL
cp /etc/fstab /etc/fstab.RAMBOOT

Edit /etc/fstab.RAMBOOT to uncomment the none root and modify the old / to /mnt/sda1. The result will look like:

Code:

UUID=5129efa8-fd32-58e4-83fd-2e3afdbcdef653 /mnt/sda1  ext4 error=remount-ro 0 1
none /       tmpfs size=95% 0 1
none /tmp     tmpfs size=95% 0 1
none /var/tmp tmpfs size=95% 0 1
none /var/log tmpfs size=95% 0 1
10.42.0.1:/netroot /netroot nfs rw,noatime,nolock,noauto 1 1

Now, make /netroot, /mnt/sda1 and mount /tmp with:

Code:

mkdir /netroot
mkdir /mnt/sda1
mount /netroot

Step 3) PREPARE CUSTOMIZED /usr/share/initramfs-tools/scripts/local

Make a copies of /usr/share/initramfs-tools/scripts/local with:

Code:

cd /usr/share/initramfs-tools/scripts/
cp -vax nfs nfs.NORMAL
cp -vax nfs nfs.RAMBOOT

Edit nfs.RAMBOOT by replacing the root mount line from this:

Code:

   nfsmount -o nolock ${roflag} ${NFSOPTS} ${NFSROOT} ${rootmnt}

to this:

Code:

mkdir /ijkijk
###nfsmount -o nolock ${roflag} ${NFSOPTS} ${NFSROOT} ${rootmnt}
   nfsmount -o nolock ${roflag} ${NFSOPTS} ${NFSROOT} /ijkijk
   mount    -t tmpfs           -o size=95%   none     ${rootmnt}
cd  ${rootmnt}
echo "Extracting from ramboot/image.tar.gz ..."
tar xzf /ijkijk/ramboot/image.tar.gz
umount /ijkijk

Step 4) CREATE CUSTOM BOOT FILES

Now, we can set up a suitable initrd for nfsramboot. Even though we're not using PXE to boot, I'm naming these with PXE to remind us that the initrd and vmlinuz are identical to those used for PXE booting.

Code:

vi /etc/initramfs-tools/initramfs.conf

Add the following line at the bottom:

Code:

BOOT=nfs

Previous versions of Debian would have a line saying BOOT=local, which you'd modify to BOOT=nfs. But not Debian 8-9. Anyway, something to look out for, perhaps, if your config files are already customized, or inherited from earlier versions, or you're using this for another Debian/Ubuntu variant.

>>> WARNING - DON'T FORGET TO COMMENT OUT BOOT=nfs IN STEP SIX <<<

Code:

cd /usr/share/initramfs-tools/scripts/
cp -vax nfs.RAMBOOT nfs
mkinitramfs -o /boot/initrd.pxe
cp -vax /boot/vmlinuz-`uname -r` /boot/vmlinuz.pxe

Step 5) CREATE THE IMAGE

We're going to temporarily move the RAMBOOT fstab in place of the NORMAL files.

Code:

apt-get clean

cp -vax /etc/fstab.RAMBOOT /etc/fstab

tar cvzf /nfsroot/ramboot/image.tar.gz --one-file-system /

cp -vax /etc/fstab.NORMAL /etc/fstab

Step 6) CLEAN UP

COMMENT OUT BOOT=nfs AT THIS TIME WITH

Code:

vi /etc/initramfs-tools/initramfs.conf

comment out the line at the bottom:

Code:

#BOOT=nfs

WARNING! If you forget to uncomment this line, then the next time a kernel upgrade is installed on the normal boot, it will create an initramfs that attempts to boot off the network instead of the local hard drive. Since the boot entry isn't set up for this, it may make the local install unbootable. Better to clean up things so the local install is in a proper state, then deal with a mystifying failure weeks or months later on.

Step 7) CREATE NETROOT GRUB ENTRY

The (possibly) final step is to set up the custom grub2 entry. You'll want to base the entry on an existing entry in /boot/grub/grub.cfg. The result will look something like this:

Code:

vi /etc/grub.d/40_custom

Code:

menuentry "Netroot" {
   echo 'Trying to boot via nfs ...'
   load_video
   insmod gzio
   if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
   insmod part_msdos
   insmod ext2
   set root='hd0,msdos1'
   echo 'Loading Linux ...'
   linux /boot/vmlinuz.pxe root=/dev/nfs nfsroot=10.42.0.1:/netroot rw  quiet
   echo 'Loading initial ramdisk ...'
   initrd /boot/initrd.pxe
}

Pay attention to "set root", because this will determine where it tries to load vmlinuz.pxe and initrd.pxe from. If you have a separate /boot partition, then you'll want to change the path of /boot/vmlinuz.pxe and /boot/initrd.pxe to /vmlinuz.pxe and /initrd.pxe.

Now change the default GRUB choice with:

Code:

vi /etc/default/grub

Code:

GRUB_DEFAULT="Netroot"

Update /boot/grub/grub.cfg with

Code:

update-grub

If you're like me, this part will take a few tries to get just right. Just boot into the local Debian install and fiddle with /etc/grub.d/40_custom until you get it right.

- - - - - 8< - - - - - cute here - - - - - 8< - - - - -

MAINTENANCE

When you want to save a new image, based on the current state, do the following while booted up into the nfsramboot client:

Code:

mount /netroot
cd /netroot/ramboot/
mv -vi image.tar.gz iback
tar cvzf /ramboot/image.tar.gz --one-file-system /

If the kernal has been upgraded, manually copy over the new initrd.img and vmlinuz with something like:

Code:

cp -vax /boot/initrd.img-3.16.0-4-686-pae /mnt/sda1/boot/initrd.pxe
cp -vax /boot/vmlinuz-3.16.0-4-686-pae /mnt/sda1/boot/vmlinuz.pxe

INSTALLING XFCE4 GUI

Here's an example for installing a GUI. It's not the most stripped down option, but it's easy to specify without being too heavyweight.

Code:
apt-get install xorg xfce4 lightdm
apt-get clean
After this, / will be about 1084MB and image.tar.gz will be about 428MB. Obviously, adding more software will increase the size of both.

Posted in Uncategorized

Views 6158 Comments 3

« Prev Main Next »

Total Comments 3

Comments

If the client has 3GB or less of RAM, and you're looking for places to shave some tmpfs drive space:

1) Move /usr/share/doc onto a local drive or nfs mount, or simply delete it entirely (software updates will tend to put stuff back into it). This can save about 100MB, which is a big deal if space is tight. According to the Debian standards, all software is required to not break if /usr/share/doc is not present.

2) Move /home/<user>/.cache/chromium and other web browser caches onto a local drive or nfs mount. A symlink will work. This can save over 100MB. Even if it's a bit slower than local ram, it's likely you'll never notice since it's still a lot faster than pulling off the internet and web browser speed tends to be limited by the speed of downloading the bits necessary off the internet anyway.

3) Move /home/<user>/.config/chromium and other web browser config onto a local drive or nfs mount. A symlink will work. This actually might not save much space, if you don't use lots of heavy plugins or whatever. But it's nice for you to be able to resume where you left off web browsing in case of power failure. Just make sure to apt to the latest version of the web browser before running it after a power failure. Otherwise, you might lose stuff if you try to run chromium with an older version than what's in the .config save (I learned this the hard way).

Moving the web browser cache/config off to an nfs share is a particularly good fit, because if the network is down you're not doing any web browsing anyway.

With these tricks, you can reduce the size of tmpfs / by well over 200MB.

Posted 07-02-2017 at 04:14 AM by IsaacKuo IsaacKuo is offline

Updated 07-02-2017 at 04:15 AM by IsaacKuo

Random note on NFS shares - it seems that in Debian 9 things can get funny if network-manager is installed and depending on the settings in /etc/network/interfaces...

Basically, resolve.conf might be broken, which may interfere with mounting nfs shares, or the interface itself might only come up too late in the boot sequence to mount the nfs shares. This can delay bootup by the 1.5 minutes it takes to timeout nfs share mounting.

The workaround I've come up with is:

1) Use apt-get to remove network-manager

Code:

apt-get remove --purge network-manager
apt-get autoremove --purge

2) create/edit /etc/resolve.conf if necessary to something like:

Code:

nameserver 192.168.1.1

Replace with your gateway IP address.

Of course, if you use /etc/hosts to specify the IP addresses of the nfs share computers, then this won't really muck up mounting of nfs shares. But name resolution is critical for accessing servers out there on the internet...

Posted 07-04-2017 at 10:11 AM by IsaacKuo IsaacKuo is offline

Alternative State-save Method

One annoying thing about NFS-RAMBOOT is that it takes a while to create a new image and it hogs network bandwidth. An alternative is to use rsync - which is very fast - to keep another copy of the "/" filetree on the server in sync with what's in RAM. Then, you run tar on the server to compress that filetree into the image tarball. This costs some extra space on the server, but it conserves network bandwidth and it keeps you going instead of waiting for the tarball to finish.

- - - - - 8< - - - - - cute here - - - - - 8< - - - - -

ON SERVER

Create the filetree folder and optionally initially populate it with:

Code:

mkdir /netroot/ramback
cd /netroot/ramback/
tar xzf /netroot/ramboot/image.tar.gz

Make a utility script to compress a new tarball for use later:

Code:

#!/bin/sh
rm -vi /netroot/ramboot64/image.tar.gz
cd /netroot/ramback/
tar cvzf /netroot/ramboot64/image.tar.gz --one-file-system .

- - - - - 8< - - - - - cute here - - - - - 8< - - - - -

ON CLIENT

Use the following command whenever you want to save the current OS state to /netroot/ramback:

Code:

rsync -vaxAX --delete /. /netroot/ramback/

If you want, you can run the tarball utility script on the server each and every time you sync up. Or you can just run the utility script once in a while. Or you can just run the utility script shortly before you intend to (re)boot the client.

The bottom line is that saving the OS state with rsync is very fast and efficient, and the tarball compression doesn't really need to be done so often. Compression is still useful, though, because it reduces client boot time and network bandwidth use.

Posted 08-04-2017 at 02:18 PM by IsaacKuo IsaacKuo is offline