[SOLVED] Problem installing Arch system on a ZFS root: "ZFS: Cannot find bootfs." on boot.

tajjada · 12-24-2013, 09:19 AM

I have been experimenting with ZFS filesystems on external HDDs for some time now to get more comfortable with using ZFS.

Today, I made my first attempt to install an Arch Linux system on a ZFS root on a USB hard drive. (My problem isn't necessarily Arch-specific though, and so I am posting here) This is mostly for educational purposes. I have many years of experience with Linux and all kinds of system configurations and software, but now is the first time I am trying out ZFS.

If everything goes well and I like how this experiment turns out, I might consider using a ZFS root for my main system over btrfs (as I am somewhat dissatisfied/unhappy with the latter).

My partition configuration (from gdisk):

Code:

Command (? for help): p
Disk /dev/sdb: 3907024896 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2FAE5B61-CCEF-4E1E-A81F-97C8406A07BB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907024862
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048          264191   128.0 MiB   8300  Linux filesystem
   3          264192      3902828543   1.8 TiB     BF00  Solaris root
   4      3902828544      3907024862   2.0 GiB     8300  Linux filesystem

Partition #1 is for grub, obviously. Partition #2 is an ext2 partition that I mount on /boot in the new system. Partition #3 is where I make my ZFS pool.
Partition #4 is an ext4 filesystem containing another minimal Arch system for recovery and setup purposes. GRUB is installed on the other system on partition #4, not in the new ZFS system.

The idea is that the kernel and initrd are in a separate boot partition so that grub does not have to deal with ZFS, and I am reusing the existing grub install from the other system.

I let grub-mkconfig generate a config file from the system on partition #4 to boot that. Then, I manually edited the generated grub.cfg file to add this menu entry for my ZFS system:

Code:

menuentry 'ZFS BOOT' --class arch --class gnu-linux --class gnu --class os {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt2'
        echo    'Loading Linux core repo kernel ...'
        linux   /vmlinuz-linux zfs=bootfs zfs_force=1 rw  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /initramfs-linux.img
}

And my ZFS configuration:

Code:

# zpool list
NAME          SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
External2TB  1.81T  6.06G  1.81T     0%  1.00x  ONLINE  -

# zpool status                                                                                                                        :(
  pool: External2TB
 state: ONLINE
  scan: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        External2TB                                                ONLINE       0     0     0
          usb-WD_Elements_1048_575836314135334C32383131-0:0-part3  ONLINE       0     0     0

errors: No known data errors

# zpool get bootfs
NAME         PROPERTY  VALUE                       SOURCE
External2TB  bootfs    External2TB/ArchSystemMain  local

# zfs list 
NAME                         USED  AVAIL  REFER  MOUNTPOINT
External2TB                 14.6G  1.77T    30K  none
External2TB/ArchSystemMain   293M  1.77T   293M  /
External2TB/PacmanCache     5.77G  1.77T  5.77G  /var/cache/pacman/pkg
External2TB/Swap            8.50G  1.78T    20K  -

The reason for the above configuration is that after I get this system to work, I want to install a second system in the same zpool on a different dataset, and have them share a pacman cache.

GRUB "boots" successfully, in that it loads the kernel and the initramfs as expected from the 2nd GPT partition. The problem is that the kernel does not load the ZFS:

Code:

ERROR: device '' not found. Skipping fsck.
ZFS: Cannot find bootfs.
ERROR: Failed to mount the real root device.
Bailing out, you are on your own. Good luck.

and I am left in busybox in the initramfs.

What am I doing wrong?

Also, here is my /etc/fstab in the new system:

Code:

# External2TB/ArchSystemMain
#External2TB/ArchSystemMain     /               zfs             rw,relatime,xattr       0 0

# External2TB/PacmanCache
#External2TB/PacmanCache        /var/cache/pacman/pkg   zfs             rw,relatime,xattr       0 0

UUID=8b7639e2-c858-4ff6-b1d4-7db9a393578f       /boot           ext4            rw,relatime     0 2

UUID=7a37363e-9adf-4b4c-adfc-621402456c55       none            swap            defaults        0 0

I also tried to boot using "zfs=External2TB/ArchSystemMain" in the kernel options, since that was the more logical way to approach my intention of having multiple systems on different datasets. It would allow me to simply create separate grub menu entries for each, with different boot datasets in the kernel parameters. I also tried setting the mount points to "legacy" and uncommenting the zfs entries in my fstab above. That didn't work either and produced the same results, and that was why I decided to try to use "bootfs" (and maybe have a script for switching between the systems by changing the ZFS bootfs and mountpoints before reboot, reusing the same grub menuentry).

I read all of the ZFS-related Arch Wiki pages before I even started with this. Followed the instructions quite closely, adapting for my scenario. Also read a lot of other articles on the internet about ZFS configurations.

I am posting here because I want to hear and learn from someone with more experience with ZFS, because the wiki pages, forum posts, and various articles don't seem to have helped me identify where my problem is.

I also posted about my problem on the Archlinux.org forums, but (in a somewhat typical fashion for that community) I was just told to RTFM and received no help. I hope this forum is more constructive.

Tell me if I need to provide anything else (other files, etc).

Thanks in advance for any help.

syg00 · 12-24-2013, 03:35 PM

/etc/mkinitcpio.conf and evidence it was rebuilt correctly with the zfs hook ?.

bigrigdriver · 12-24-2013, 03:41 PM

Quote:

Then, I manually edited the generated grub.cfg file to add this menu entry for my ZFS system:

This isn't the correct way to update grub to include a new bootable OS to grub.cfg. As root, run /usr/sbin/update-grub. That will run a probe routine which will find the new OS and make a menu entry for it.

If you need to make other changes to grub.cfg, you may also edit /etc/default/grub, then run update-grub after making changes.

I hope this helps you.

syg00 · 12-24-2013, 05:56 PM

update-grub is a Ubuntu-ism that Debian picked up, it is not part of upstream grub2. Likewise os-prober may or may not be present in a non_Debian derived distro.

tajjada · 12-25-2013, 02:07 AM

Yes, I have the zfs hook in mkinitcpio.conf right before filesystems and after keyboard. If I didn't have it, I would not be getting the error message I am getting on boot, as there would be no ZFS

.

And yes, I am aware that the "recommended" way to modify the GRUB configuration for new operating systems is using autogeneration commands ("grub-mkconfig" and such), but those do not detect ZFS, and in my case there is a separate /boot partition outside of the ZFS, which could just as well belong to any OS. No way for a script to know what the root fs is. Because of that, grub-mkconfig did not detect the new system, and I had to add the entry manually. The fault does not seem to be with GRUB, as it loads the kernel and initramfs just fine. Looks like the problem is in my ZFS setup, and maybe the zfs kernel parameter I am passing to the kernel.

Anyway, disregard this. Spent too much time trying to figure out why it doesn't work. I will reformat that hard drive and try something else, I'll keep experimenting with ZFS on virtual machine images when I have time. Try a simpler setup first, maybe.

Thank you for your input anyway.