Arch Linux

ckujau

Member

Registered: 2017-02-02

Posts: 11

Xen/PVH unable to boot with recent kernels

So, this Xen/PVH DomU has been running the linux-hardened kernel for a while now, but after upgrading from 6.8.7-hardened1 to 6.8.9-hardened1 the DomU is no longer able to access its root disk:

Loading Linux linux-hardened ...
     [ vmlinuz-linux-harden  11.35MiB  100%  7.30MiB/s ]Loading initial ramdisk ...
     [ initramfs-linux-hard  8.62MiB  100%  177.72KiB/s ]:: running early hook [udev]
Starting systemd-udevd version 256.1-1-arch-g34ba18b^
:: running hook [udev]
:: Triggering uevents...
:: running hook [keymap]
:: Loading keymap...kbd_mode: KDSKBMODE: Inappropriate ioctl for device
done.
Waiting 10 seconds for device /dev/disk/by-uuid/8eb52bf7-d289-491d-a9c0-6175d577ae99 ...
Waiting 10 seconds for device /dev/disk/by-uuid/8eb52bf7-d289-491d-a9c0-6175d577ae99 ...
ERROR: device 'UUID=8eb52bf7-d289-491d-a9c0-6175d577ae99' not found. Skipping fsck.
:: mounting 'UUID=8eb52bf7-d289-491d-a9c0-6175d577ae99' on real root
mount: /new_root: can't find UUID=8eb52bf7-d289-491d-a9c0-6175d577ae99.
ERROR: Failed to mount 'UUID=8eb52bf7-d289-491d-a9c0-6175d577ae99' on real root
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off

[rootfs ~]# cat /proc/partitions 
major minor  #blocks  name

 202        0   15728640 xvda
 202        1      64000 xvda1
 202        2    1835008 xvda2
 202        3      66944 xvda3

So, this looks OK, and I can read the block devices via less(1), but mounting still fails:

[rootfs ~]# mount -t ext4 /dev//xvda1 /new_root
mount: /new_root: can't read superblock on /dev/xvda1.

[rootfs ~]# blkid
/dev/xvda2: PARTUUID="67ded137-02"
/dev/xvda3: PARTUUID="67ded137-03"
/dev/xvda1: PARTUUID="67ded137-01"

Notice how the UUID label cannot be read from the xvda partitions.

The same happens with the "linux" package (vanilla kernel), but luckily I had the linux-lts kernel (6.6.34-1-lts) installed and this is booting just fine and the UUID labels can be read just fine.

So, I wanted to find out what changed, but the kernel configuration is not available from the package file, so I used extract-ikconfig to extract the kernel configuration from a non-running linux-6.9.5-arch1 kernel and compared it to the configuration from 6.6.34-1-lts. Searching for "xen" pointed me to this:

-CONFIG_XEN_PVCALLS_BACKEND=y
+CONFIG_XEN_PVCALLS_BACKEND=m
 CONFIG_XEN_SCSI_BACKEND=m
 CONFIG_XEN_PRIVCMD=m
-CONFIG_XEN_PRIVCMD_IRQFD=y
+CONFIG_XEN_PRIVCMD_EVENTFD=y

(i.e. 6.6.34-1-lts still had CONFIG_XEN_PVCALLS_BACKEND compiled in, while linux-6.9.5-arch1 doesn't), but that doesn't appear to be neccessary to be able to boot domains.

Before digging further, does anybody else have Xen PVH domains running and experiences maybe similar issues?