You are not logged in.

#1 2013-12-24 12:26:42

Registered: 2013-12-24
Posts: 5

[SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

I have been experimenting with ZFS filesystems on external HDDs for some time now to get more comfortable with using ZFS in the hopes of one day reinstalling my system on a ZFS root.

Today, I tried installing a system on an USB external HDD, as my first attempt to install on ZFS (I wanted to try in a safe, disposable environment before I try this on my main system).

My partition configuration (from gdisk):

Command (? for help): p
Disk /dev/sdb: 3907024896 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2FAE5B61-CCEF-4E1E-A81F-97C8406A07BB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907024862
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048          264191   128.0 MiB   8300  Linux filesystem
   3          264192      3902828543   1.8 TiB     BF00  Solaris root
   4      3902828544      3907024862   2.0 GiB     8300  Linux filesystem

Partition #1 is for grub, obviously. Partition #2 is an ext2 partition that I mount on /boot in the new system. Partition #3 is where I make my ZFS pool.
Partition #4 is an ext4 filesystem containing another minimal Arch system for recovery and setup purposes. GRUB is installed on the other system on partition #4, not in the new ZFS system.

I let grub-mkconfig generate a config file from the system on partition #4 to boot that. Then, I manually edited the generated grub.cfg file to add this menu entry for my ZFS system:

menuentry 'ZFS BOOT' --class arch --class gnu-linux --class gnu --class os {
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt2'
        echo    'Loading Linux core repo kernel ...'
        linux   /vmlinuz-linux zfs=bootfs zfs_force=1 rw  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /initramfs-linux.img

My ZFS configuration:

# zpool list
External2TB  1.81T  6.06G  1.81T     0%  1.00x  ONLINE  -

# zpool status                                                                                                                        :(
  pool: External2TB
 state: ONLINE
  scan: none requested

        NAME                                                       STATE     READ WRITE CKSUM
        External2TB                                                ONLINE       0     0     0
          usb-WD_Elements_1048_575836314135334C32383131-0:0-part3  ONLINE       0     0     0

errors: No known data errors

# zpool get bootfs
NAME         PROPERTY  VALUE                       SOURCE
External2TB  bootfs    External2TB/ArchSystemMain  local

# zfs list 
NAME                         USED  AVAIL  REFER  MOUNTPOINT
External2TB                 14.6G  1.77T    30K  none
External2TB/ArchSystemMain   293M  1.77T   293M  /
External2TB/PacmanCache     5.77G  1.77T  5.77G  /var/cache/pacman/pkg
External2TB/Swap            8.50G  1.78T    20K  -

The reason for the above configuration is that after I get this system to work, I want to install a second system in the same zpool on a different dataset, and have them share a pacman cache.

GRUB "boots" successfully, in that it loads the kernel and the initramfs as expected from the 2nd GPT partition. The problem is that the kernel does not load the ZFS:

ERROR: device '' not found. Skipping fsck.
ZFS: Cannot find bootfs.
ERROR: Failed to mount the real root device.
Bailing out, you are on your own. Good luck.

and I am left in busybox in the initramfs.

What am I doing wrong?

Also, here is my /etc/fstab in the new system:

# External2TB/ArchSystemMain
#External2TB/ArchSystemMain     /               zfs             rw,relatime,xattr       0 0

# External2TB/PacmanCache
#External2TB/PacmanCache        /var/cache/pacman/pkg   zfs             rw,relatime,xattr       0 0

UUID=8b7639e2-c858-4ff6-b1d4-7db9a393578f       /boot           ext4            rw,relatime     0 2

UUID=7a37363e-9adf-4b4c-adfc-621402456c55       none            swap            defaults        0 0

I also tried to boot using "zfs=External2TB/ArchSystemMain" in the kernel options, since that was the more logical way to approach my intention of having multiple systems on different datasets. It would allow me to simply create separate grub menu entries for each, with different boot datasets in the kernel parameters. I also tried setting the mount points to "legacy" and uncommenting the zfs entries in my fstab above. That didn't work either and produced the same results, and that was why I decided to try to use "bootfs" (and maybe have a script for switching between the systems by changing the ZFS bootfs and mountpoints before reboot, reusing the same grub menuentry).

Thanks in advance for any help.

Last edited by tajjada (2013-12-30 20:03:09)


#2 2013-12-24 12:39:31

From: :wq
Registered: 2008-12-01
Posts: 9,438

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

Can't help you with a ZFS root.  There is a wiki page on the topic.  My 2 cents: don't use ZFS for your root.

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs • 1.21 JIGOWATTS!


#3 2013-12-24 12:48:12

Registered: 2013-12-24
Posts: 5

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

There is a wiki page on the topic.

I read all of them wiki pages on ZFS before I even started. Followed the instructions quite closely, adapting for my scenario. Don't tell me to RTFM, because I did. Read a lot of other articles on the internet too. For everything, not just Arch, forums are the last place I go to when I need help. I have set up and used arch systems on many different configurations for a couple of years now, and today was the first time I felt the need to register on the forums to post...

I am posting here because I want to hear and learn from someone with more experience with ZFS, because the wiki pages, forum posts, and various articles don't seem to have helped me identify where my problem is.

Can't help you with a ZFS root.

Well, then keep quiet. I want to hear input from someone who can. Don't troll.

Last edited by tajjada (2013-12-24 12:56:09)


#4 2013-12-29 14:55:33

Registered: 2007-12-30
Posts: 29

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

Sounds like a zpool.cache issue. I'm guessing your zpool.cache inside your arch-chroot is not up to date. So on boot the ZFS hook cannot find the bootfs. At least, that's what I assume the issue is, because of this line:

ERROR: device '' not found. Skipping fsck.

If your zpool.cache was populated, it would spit out something other than an empty string.

Some assumptions:
- You're using the ZFS packages provided by demizer (repository or AUR).
- You're using the Arch Live ISO or some version of it.

On cursory glance your configuration looks good. But verify anyway. Here are the steps you should follow to make sure your zpool.cache is correct and up to date:

Outside arch-chroot:
- Import pools (not using '-R') and verify the mountpoints.
- Make a copy of the /etc/zfs/zpool.cache before you export any pools. Again, make a copy of the /etc/zfs/zpool.cache before you export any pools. The reason for this is once you export a pool the /etc/zfs/zpool.cache gets updated and removes any reference to the exported pool. This is likely the cause of your issue, as you would have an empty zpool.cache.
- Import the pool containing your root filesystem using the '-R' flag, and mount /boot within.
- Make sure to copy your updated zpool.cache to your arch-chroot environment.

Inside arch-chroot:
- Make sure your bootloader is configured properly (i.e. read 'mkinitcpio -H zfs').
- Use the 'udev' hook and not the 'systemd' one in your mkinitcpio.conf. The zfs-utils package does not have a ported hook (as of 0.6.2_3.12.6-1).
- Update your initramfs.

Outside arch-chroot:
- Unmount filesystems.
- Export pools.
- Reboot.

Inside new system:
- Make sure to update the hostid then rebuild your initramfs. Then you can drop the 'zfs_force=1'.

Good luck. I enjoy root on ZFS myself. However, I wouldn't recommend swap on ZFS. Despite what the ZoL tracker says, I still ran into deadlocks on occasion (as of a month ago). However, I cannot say definitely the cause of the issue; but it resolved when I moved swap off ZFS to a dedicated partition.

Last edited by NVS (2013-12-29 14:56:44)


#5 2013-12-29 16:38:27

Registered: 2013-12-24
Posts: 5

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

YES! Thank you.

I have no idea what exactly it was that I did that fixed it, but I repartitioned and reformatted the external HDD and started all over again, and made sure to go through your checklist above. I have a feeling that it was the zpool.cache that was causing the problems though (like you suggested).

I am writing this reply from a full KDE desktop running off my external HDD on a ZFS root. GREAT SUCCESS!!

Last edited by tajjada (2013-12-30 20:14:53)


#6 2013-12-29 19:13:06

Registered: 2007-12-30
Posts: 29

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

Glad to here. If that addressed your issue (sounds like it did), then you can mark this as [SOLVED].


#7 2013-12-30 20:05:53

Registered: 2013-12-24
Posts: 5

Re: [SOLVED] Installing on ZFS root: "ZFS: cannot find bootfs" on boot.

Oh OK, I just wasn't aware that on the Arch forums one has to edit the title to manually mark as [SOLVED]. I looked for a button or something like on other forums, and after not finding one I just assumed that I wasn't supposed to mark my thread as such.

A quick google search led me to the forum thread about what I should do.

Again, thank you for the heads up!


Board footer

Powered by FluxBB