Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

kevdog · 2020-02-02 14:25:50

I'm sorry if this is the wrong forum for this question, however just looking for guidance

My host system is FreeNAS which creates VM's with the bhyve hypervisor. The VM was created with a UEFI boot using systemd-boot. Partition structure:

lsblk                                                                 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   40G  0 disk
├─sda1   8:1    0  512M  0 part /boot
└─sda2   8:2    0 39.5G  0 part

Within FreeNAS I created an Arch Linux VM with ZFS on Root. VM boots and I'm overall pleased.

I wanted to duplicate or clone this VM to avoid setting things up all over again.

Within FreeNAS, I snapshotted my installation, then cloned via a zfs send/zfs receive. I then had to recreate a new VM but used the new cloned dataset as the data source. Unfortunately this means the new VM was created with all new hardware IDs.

I then went to boot "the new Arch" installation however ran into a problem:

ZFS: importing pool tank
cannot import 'tank': no such pool or dataset

I thought this was strange so I booted the VM using the remastered archiso with the zfs modules

Once booting the archiso, I saw the disk partition structure was preserved.

Following the instructions on the wiki (but wiki didn't exactly address this problem):

modprobe zfs
zpool import -a -R /mnt
mount /dev/sda2 /mnt/boot
arch-chroot /mnt /bin/bash

Once inside the arch-chroot, the zfs datasets were appropriately preserved

I tried exporting/reimporting pool and then rebooting but faced same problem with not being able to find the tank partition

What exactly do I do at this point? Generate initramfs or something different?

In creating the new VM, the hardware ID's have changed (which might be part of problem)??
(old installation)

zpool status                                                         
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:00:04 with 0 errors on Sat Feb  1 09:42:25 2020
config:

	NAME                                              STATE     READ WRITE CKSUM
	tank                                              ONLINE       0     0     0
	  ata-BHYVE_SATA_DISK_BHYVE-866B-88CE-A25F-part2  ONLINE       0     0     0

(new installation)

# zpool status                                                                    :(
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:00:04 with 0 errors on Sat Feb  1 15:42:25 2020
config:

	NAME                                              STATE     READ WRITE CKSUM
	tank                                              ONLINE       0     0     0
	  ata-BHYVE_SATA_DISK_BHYVE-34DB-4C4D-2B38-part2  ONLINE       0     0     0

WorMzy · 2020-02-02 14:31:11

Mod note: Moving to AUR Issues.

kevdog · 2020-02-02 19:42:41

I'm going to answer my own question here hoping it will help someone else (or probably me) when they encounter this error in the future

Conclusion: the initramfs needs to be regenerated

Steps to do this:
1. Please use alternative media to boot to VM (ie remastered Arch Linux ISO with the zfs modules). The remastered archiso is probably the same media you used to recreate ZFS on root for Arch Linux. Details how to generate this specific archiso are: https://wiki.archlinux.org/index.php/ZF … an_archiso

2. (Optional step -- I prefer this step since without it I normally have to log into VM through VNC). Once booted you will be automatically logged into the archiso as root. Please set passwd for root and and optionally enable the sshd so you can log into the system via alternative or normal means

passwd
systemctl start sshd.service

Login to system from remote ssh terminal as root

3. Manually import and mount the zpool(s), and manually mount the /boot partition (if required)

First confirm the partitions for your cloned VM exist and seem to be correct. An example shown below shows two partitions sda1, sda2. When created sda1 was the /boot partition and sda2 was the main zpool partition.

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 546.8M  1 loop /run/archiso/sfs/airootfs
sda      8:0    0    40G  0 disk
├─sda1   8:1    0   512M  0 part
└─sda2   8:2    0  39.5G  0 part
sr0     11:0    1   661M  0 rom  /run/archiso/bootmnt

Find the ID# of the zpool throught zpool import: (In example below the id# = 12523338641105440463)

root@archiso ~ # zpool import
   pool: tank
     id: 12523338641105440463
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	tank                                              ONLINE
	  ata-BHYVE_SATA_DISK_BHYVE-34DB-4C4D-2B38-part2  ONLINE

Import the zpool manually and perform a mount. In the example below the zpool is known as "tank". Also mount the /boot partition if you originally configured your setup with a boot partition. Mount any other partitions your setup requires.

zpool import 12523338641105440463 -d /dev/disk/by-id -R /mnt tank
mount /dev/sda1 /mnt/boot

If needed you can verify correct mounting of the zpool with "zfs mount" and/or "zpool status"

3a. Bring the zpool.cache file into your new system

In the example below, the zpool on root is known as tank. Please adjust accordingly to your zpool name

# cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

if you do not have /etc/zfs/zpool.cache, create it:

# zpool set cachefile=/etc/zfs/zpool.cache tank

4.chroot into installation to regenerate the hostid and initial ramdisk

arch-chroot /mnt /bin/bash
mv /etc/hostid /etc/hostid.bak
zgenhostid $(hostid)

My /boot/loader/entries/arch.conf file appears as the following:

title         Arch Linux↲
linux       /vmlinuz-linux↲
initrd       /initramfs-linux.img↲
options   zfs=tank/ROOT/default rw↲
options   zfs_import_dir=/dev/disk/by-id↲
options...zfs_force=1↲

I'm not sure if the zfs_force=1 is necessary.

mkinitcpio -p linux
exit

5. Unmount all partitions and export the zpool. In the example below the name of the zpool=tank. Adjust accordingly.

unmount /mnt/boot
zfs umount -a
zpool export tank

6. Eject archiso boot media and reboot VM. It should boot cleanly at this point however networking may not be functioning.

7. Additional steps that might be needed

- Rename the hostid (newarchlinuxvm=new hostid)

   hostnamectl set-hostname newarchlinuxvm

- Network Parameter Adjustment: Likely needed due to MAC address of networking card changed when migrating VMs. It's difficult for me to make exact recommendations on what to do here, however I'll give an example below. Please note that I used systemd-networkd to manage my network interfaces. If you use a different network manager as discussed here: https://wiki.archlinux.org/index.php/Ne … figuration, please consult the appropriate Arch Wiki page for assistance.

The MAC address can be discovered with the ip link command. Below the MAC address is shown as: 00:a0:98:34:70:23

#ip link                                                    
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:a0:98:34:70:23 brd ff:ff:ff:ff:ff:ff

My configuration file for systemd-networkd is the following:

cat /etc/systemd/network/20-wired.network                  
[Match]
Name=eth0

[Network]
DHCP=ipv4
DNS=10.0.1.1

To modify the MAC address associated with eth0, I need to modify /etc/udev/rules.d/10-network.rules for the following (this file needed to be generated from scratch -- it does not exist in a base system)

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:a0:98:34:70:23", NAME="eth0"

8. Reboot and enjoy!. The beauty of working with a hypervisor is its possible to create and setup a base system once (create the VM), then clone the base setup (clone VM) as many times as needed and setup each VM accordingly. This setup explained above is applicable to VMs that were created with the ZFS file system on root. With these particular systems, the initramfs needs to be regenerated appropriately to boot the system.

Last edited by kevdog (2020-02-10 04:18:40)

malbeth · 2020-08-19 00:37:56

How did you even manage to install it in the first place?! I've been looking everywhere and trying anything but I can't find a single source of information on how to do that.

I've create several VMs in FreeNAS and each time if I select an Arch Linux ISO as installation medium I just get a blank screen in the Serial view after the boot menu if I use UEFI.

If I use UEFI-CSM then I get the loading Linux imgs but nothing else after that... Same thing with the ZFS remastered iso...

Would really love to know how you managed to install it!

Thanks!

malbeth · 2020-08-20 01:51:48

Ok I finally found out how to do boot into the live CD from this post! http://freebsd.1045724.x6.nabble.com/Ar … 55234.html

You have to tweak the ISO to boot on tty0!

Arch Linux

#1 2020-02-02 14:25:50

Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

#2 2020-02-02 14:31:11

Re: Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

#3 2020-02-02 19:42:41

Re: Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

#4 2020-08-19 00:37:56

Re: Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

#5 2020-08-20 01:51:48

Re: Cloned Arch Linux ZFS Root Installation - Now I can't boot the clone

Board footer