Archlinux libvirt - some cloud images not running correctly with UEFI

stiw47 · 2024-08-19 08:54:52

Hi. Could someone help me troubleshoot this? I have 2 different hardware hosts, with Archlinux as host OS. I am using KVM/QEMU/libvirt for virtualization, and I am managing VMs from Cockpit UI on both hosts. Management is done from Cockpit 95% of the time, and very rare I using virsh, mostly if need to troubleshoot something. I am using default NAT network in libvirt on both hosts, and would like to avoid bridge as for now. In general, everything with libvirt is very close to vanilla setup.

On host #1, hostname archmedia, every cloud image I spin up by using UEFI firmware, get automatically IP assigned from libvirt DHCP, and it's default range 192.168.122.0/24. Also, basic cloud-init configuration, such as root password and additional username:password are working - I am able to login with both credentials when VM is up.
On host #2, hostname archvideo, also wehn using UEFI firmware, NOT every cloud image get automatically IP assigned from libvirt DHCP. IP range is the same - 192.168.122.0/24, and some cloud images are not getting IP. I even cannot assign IP manually through # virsh net-edit default, by adding e.g. following:

    <dhcp>
      <range start='192.168.122.100' end='192.168.122.254'/>
      <host mac='52:54:00:99:5f:a2' name='ubuntu-24-10' ip='192.168.122.50'/>
    </dhcp>

Also, on cloud images where I am not getting IP assigned, same cloud-init configuration as described for host #1, is not working - I am not able to login to VM, once it is up.

Since Archlinux is rolling distro, basically it is enough that you do:

# pacman -Syu

(^^which was already done on both hosts), and to, let's say, have 2 same OSs, with the difference of installed packages of course.

Host #1 archmedia, OS, kernel and CPU:

[stiw47@archmedia ~]$ archfetch
      ___     stiw47@archmedia
     (.· |    OS:         Arch Linux
     (<> |    KERNEL:     Linux 6.10.5-lqx1-1-lqx
    / __  \   UPTIME:     2 minutes
   ( /  \ /|  PACKAGES:   1717
  _/\ __)/_)  SHELL:      bash
  \/-____\/   CPU:        Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
              GPU:        Quadro K600
Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller

Host #2 archvideo, OS, kernel and CPU:

[stiw47@archvideo ~]$ archfetch
      ___     stiw47@archvideo
     (.· |    OS:         Arch Linux
     (<> |    KERNEL:     Linux 6.10.5-lqx1-1-lqx
    / __  \   UPTIME:     26 minutes
   ( /  \ /|  PACKAGES:   1860
  _/\ __)/_)  SHELL:      bash
  \/-____\/   CPU:        AMD Ryzen 5 3600 6-Core Processor
              GPU:        GeForce GTX 1650

Some, in my opinion, relevant packages:

Host #1 archmedia:

[stiw47@archmedia ~]$ pacman -Q | grep -E 'virt|firmware|qemu|linux'
archlinux-appstream-data 20240526-1
archlinux-keyring 20240709-1
libgovirt 2:0.3.9-2
libvirt 1:10.6.0-1
libvirt-dbus 1.4.1-3
libvirt-glib 5.0.0-1
libvirt-python 1:10.6.0-1
linux 6.10.5.arch1-1
linux-api-headers 6.10-1
linux-firmware 20240809.59460076-1
linux-firmware-whence 20240809.59460076-1
linux-headers 6.10.5.arch1-1
linux-lqx 6.10.5.lqx1-1
linux-lqx-headers 6.10.5.lqx1-1
qemu-base 9.0.2-1
qemu-common 9.0.2-1
qemu-img 9.0.2-1
qemu-system-x86 9.0.2-1
qemu-system-x86-firmware 9.0.2-1
qt5-virtualkeyboard 5.15.14-1
qt6-virtualkeyboard 6.7.2-1
util-linux 2.40.2-1
util-linux-libs 2.40.2-1
virt-install 4.1.0-3
virt-viewer 11.0-3
virtiofsd 1.11.1-1

Host #2 archvideo:

[stiw47@archvideo ~]$ pacman -Q | grep -E 'virt|firmware|qemu|linux'
archlinux-appstream-data 20240526-1
archlinux-keyring 20240709-1
lib32-util-linux 2.40.2-1
libgovirt 2:0.3.9-2
libvirt 1:10.6.0-1
libvirt-dbus 1.4.1-3
libvirt-glib 5.0.0-1
libvirt-python 1:10.6.0-1
libvirt-storage-gluster 1:10.6.0-1
libvirt-storage-iscsi-direct 1:10.6.0-1
linux 6.10.5.arch1-1
linux-api-headers 6.10-1
linux-firmware 20240809.59460076-1
linux-firmware-whence 20240809.59460076-1
linux-headers 6.10.5.arch1-1
linux-lqx 6.10.5.lqx1-1
linux-lqx-headers 6.10.5.lqx1-1
qemu-audio-alsa 9.0.2-1
qemu-audio-dbus 9.0.2-1
qemu-audio-jack 9.0.2-1
qemu-audio-oss 9.0.2-1
qemu-audio-pa 9.0.2-1
qemu-audio-pipewire 9.0.2-1
qemu-audio-sdl 9.0.2-1
qemu-audio-spice 9.0.2-1
qemu-base 9.0.2-1
qemu-block-curl 9.0.2-1
qemu-block-dmg 9.0.2-1
qemu-block-nfs 9.0.2-1
qemu-block-ssh 9.0.2-1
qemu-chardev-spice 9.0.2-1
qemu-common 9.0.2-1
qemu-desktop 9.0.2-1
qemu-emulators-full 9.0.2-1
qemu-hw-display-qxl 9.0.2-1
qemu-hw-display-virtio-gpu 9.0.2-1
qemu-hw-display-virtio-gpu-gl 9.0.2-1
qemu-hw-display-virtio-gpu-pci 9.0.2-1
qemu-hw-display-virtio-gpu-pci-gl 9.0.2-1
qemu-hw-display-virtio-vga 9.0.2-1
qemu-hw-display-virtio-vga-gl 9.0.2-1
qemu-hw-s390x-virtio-gpu-ccw 9.0.2-1
qemu-hw-usb-host 9.0.2-1
qemu-hw-usb-redirect 9.0.2-1
qemu-hw-usb-smartcard 9.0.2-1
qemu-img 9.0.2-1
qemu-pr-helper 9.0.2-1
qemu-system-aarch64 9.0.2-1
qemu-system-alpha 9.0.2-1
qemu-system-alpha-firmware 9.0.2-1
qemu-system-arm 9.0.2-1
qemu-system-arm-firmware 9.0.2-1
qemu-system-avr 9.0.2-1
qemu-system-cris 9.0.2-1
qemu-system-hppa 9.0.2-1
qemu-system-hppa-firmware 9.0.2-1
qemu-system-loongarch64 9.0.2-1
qemu-system-m68k 9.0.2-1
qemu-system-microblaze 9.0.2-1
qemu-system-microblaze-firmware 9.0.2-1
qemu-system-mips 9.0.2-1
qemu-system-nios2 9.0.2-1
qemu-system-or1k 9.0.2-1
qemu-system-ppc 9.0.2-1
qemu-system-ppc-firmware 9.0.2-1
qemu-system-riscv 9.0.2-1
qemu-system-riscv-firmware 9.0.2-1
qemu-system-rx 9.0.2-1
qemu-system-s390x 9.0.2-1
qemu-system-s390x-firmware 9.0.2-1
qemu-system-sh4 9.0.2-1
qemu-system-sparc 9.0.2-1
qemu-system-sparc-firmware 9.0.2-1
qemu-system-tricore 9.0.2-1
qemu-system-x86 9.0.2-1
qemu-system-x86-firmware 9.0.2-1
qemu-system-xtensa 9.0.2-1
qemu-tools 9.0.2-1
qemu-ui-curses 9.0.2-1
qemu-ui-dbus 9.0.2-1
qemu-ui-egl-headless 9.0.2-1
qemu-ui-gtk 9.0.2-1
qemu-ui-opengl 9.0.2-1
qemu-ui-sdl 9.0.2-1
qemu-ui-spice-app 9.0.2-1
qemu-ui-spice-core 9.0.2-1
qemu-user 9.0.2-1
qemu-vhost-user-gpu 9.0.2-1
qt6-virtualkeyboard 6.7.2-1
syslinux 6.04.pre2.r11.gbf6db5b4-4
util-linux 2.40.2-1
util-linux-libs 2.40.2-1
virt-install 4.1.0-3
virt-viewer 11.0-3
virtiofsd 1.11.1-1
whatsapp-for-linux-debug 1.6.5-1

TBH, I didn't realized before that I somehow installed a looot more qemu/libvirt packages on the host where I have a problem, and I see it now . I am pretty sure that most of them are not needed for my basic workload with VMs, but also not sure could this cause the problem.

Anyway, since I honestly don't know which info needs to provide, I would stop to pasting here probably un-relevant info and garbage, and would ask you guys how to troubleshoot this.

Certain example - let's take 2 exact cloud images:
1. Arch-Linux-x86_64-cloudimg-20240815.255016.qcow2 - https://gitlab.archlinux.org/archlinux/ … 1/download
2. oracular-server-cloudimg-amd64.img - https://cloud-images.ubuntu.com/oracula … -amd64.img

Both of them are working as expected with UEFI firmware chosen in Cockpit (libvirt) on Host #1 archmedia - getting IP address automatically and I am able to login with configured cloud-init users/passwords.

Only Archlinux is working as expected with UEFI firmware chosen in Cockpit (libvirt) on Host #2 archvideo - same as described above
Ubuntu neither getting IP, nor I am able to login with configured cloud-init users/passwords, if I spin up it with UEFI firmware. If I spin up same this Ubuntu with BIOS firmware, everything is working as expected.
I also noticed it is the same case on host #2 for same type of cloud image of Ubuntu 20.04 (but again, all these images are working correctly on host #1 with UEFI firmware).
I also noticed in VNC console of the Cockpit UI, that during VM boot of Ubuntu oracular-server-cloudimg-amd64.img on Host #2 (problematic one), boot log stuck for few minutes on this part, before it gives login prompt:

But I don't understand if could be concluded from here what the problem could be. This "stuck" is not the case on the Host #1

Apologize if something is not formatted correctly, and if I did not provided enough info. The thing is I am not sure what to provide, and would be glad to hear additional info request. Also, since this happens only when spin up VMs with UEFI firmware, don't know if it could be something simple, which I did not enabled e.g. in BIOS of the HW machine Host #2, or something similar.

Many thanks upfront.

stiw47 · 2024-08-19 10:54:41

Intermediate update, it is not related with lot of unnecessary packages on host #2. I uninstalled them:

Host #1:

[stiw47@archmedia ~]$ pacman -Q | grep qemu
qemu-base 9.0.2-1
qemu-common 9.0.2-1
qemu-img 9.0.2-1
qemu-system-x86 9.0.2-1
qemu-system-x86-firmware 9.0.2-1

Host #2:

[stiw47@archvideo ~]$ pacman -Q | grep qemu
qemu-base 9.0.2-1
qemu-common 9.0.2-1
qemu-img 9.0.2-1
qemu-system-x86 9.0.2-1
qemu-system-x86-firmware 9.0.2-1

Host #1:

[stiw47@archmedia ~]$ pacman -Q | grep virt
libgovirt 2:0.3.9-2
libvirt 1:10.6.0-1
libvirt-dbus 1.4.1-3
libvirt-glib 5.0.0-1
libvirt-python 1:10.6.0-1
qt5-virtualkeyboard 5.15.14-1
qt6-virtualkeyboard 6.7.2-1
virt-install 4.1.0-3
virt-viewer 11.0-3
virtiofsd 1.11.1-1

Host #2:

[stiw47@archvideo ~]$ pacman -Q | grep virt
libgovirt 2:0.3.9-2
libvirt 1:10.6.0-1
libvirt-dbus 1.4.1-3
libvirt-glib 5.0.0-1
libvirt-python 1:10.6.0-1
qt6-virtualkeyboard 6.7.2-1
virt-install 4.1.0-3
virt-viewer 11.0-3
virtiofsd 1.11.1-1

Still the same.

Arch Linux

#1 2024-08-19 08:54:52

Archlinux libvirt - some cloud images not running correctly with UEFI

#2 2024-08-19 10:54:41

Re: Archlinux libvirt - some cloud images not running correctly with UEFI

Board footer