Arch Linux

Simulacrum

Member

Registered: 2017-02-23

Posts: 28

Boot stucks when adding vfio-pci IDs

Hey everyone

Im on a fresh Arch installation and followed the steps described in PCI passthrough via OVMF. Sadly, when i add die VFIO PCI IDs, the system doesnt boot anymore (every other step from the instruction seem to work).

The last thing my monitor shows when trying to boot is:

:: running early hook [udev]
Starting version  251.6-1-arch

Quick facts:

- The device is a NVIDIA GeForce GTX 1070
- I checked journalctl which has not a record for that boot attempt (which is not really suprinsingly).
- Removing the IDs from the kernel parameters and regenerating the GRUB config fixes the problem. Also, GRUB in recovery   mode boots (but loads a Nouveau driver to my pci device)
- I'm currently running

 uname -a
Linux machine 6.0.1-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 13 Oct 2022 18:58:49 +0000 x86_64 GNU/Linux 

snippets from the /etc/mkinitcpio.conf

...
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
...
HOOKS=(base udev autodetect modconf block encrypt filesystems keyboard fsck)
...

snippets from my /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX="cryptdevice=/dev/sdb2:cryptroot"
...
# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=false

I had it working on my installation before, doing the exact same things as this time (as far as i can recall it). If you need any further information, I'd be happy to provide it.

Best regards and thanks in advance,
S.

Simulacrum

Member

Registered: 2017-02-23

Posts: 28

Re: Boot stucks when adding vfio-pci IDs

Quick Update:

I fixed it by downgrading the packages to the 01.10.2022 by using the following Server in /etc/pacman.d/mirrorlist:

Server=https://archive.archlinux.org/repos/2022/10/01/$repo/os/$arch

So its probably a bug in one of the packages. Sadly, I do not have the time to dive deeper in the investigation.

TieMajor

Member

Registered: 2022-02-04

Posts: 3

Re: Boot stucks when adding vfio-pci IDs

I had the same issue expect my system was actually booting. It seems only the framebuffer was messed up (I am not using the quiet option). I am using linux-vfio as my kernel and updating to 6.0.2 is was caused the issue for me. I looked at older kernel logs to see if there was any difference but nothing jumped to me.

tsipizic

Member

Registered: 2015-05-13

Posts: 3

Re: Boot stucks when adding vfio-pci IDs

I have the same setup and same issue preventing boot.

Edit: Funny thing. Boot is not broken but if you have full disk encryption the password prompt is not visible. If you blindly type the password boot works and wayland starts normally
https://forum.level1techs.com/t/linux-k … ugh/190039

Last edited by tsipizic (2022-10-18 12:42:23)

Murray7501

Member

Registered: 2022-08-10

Posts: 1

Re: Boot stucks when adding vfio-pci IDs

What I do is :
Don't load 'vfio-pci', 'nouveau' , 'nvidia*' kernel modules on boot.
Use a small script I found on the web, to get the 'ids=' values for vfio-pci.
-- script to get ids values --
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
--
Run the below script as root.
-- script that loads kernel modules & starts lxdm --
#!/bin/sh
# change ids= to match the passthrough GPU
modprobe vfio-pci ids=xxxx:yyyy,xxxx:yyyy
modprobe nvidia
modprobe nvidia-uvm
modprobe nvidia-drm
systemctl start lxdm & exit
--
!! Downside you lose output from the normal console until you reboot when vfio-pci is loaded !!
If someone knows a way to disable the framebuffer in UEFI-GRUB and it works around this let me know.
Hope this helps someone until fixed --

-- edit : I've looked at this a bit closer and for me its the efi-framebuffer that dies
            as evident by looking in /sys/bus/platform/drivers/efi-framebuffer directory after starting up X and
            then use a terminal emulator like urxvt, xterm etc. to have a look in that directory.

Last edited by Murray7501 (2022-10-21 07:22:51)

SteamedHams

Member

Registered: 2022-10-22

Posts: 1

Re: Boot stucks when adding vfio-pci IDs

I found a fix (or maybe, workaround) for this in my case where I use the Intel iGPU as primary CPU -- the problem seems to be that vfio-pci kills the EFI framebuffer indeed, and boot works after you blindly enter the password as once root is mounted i915 module is loaded.

You can solve this by making i915 module be loaded before vfio-pci on mkinitcpio.conf:

MODULES=(i915 vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

This makes the early password prompt be visible using the i915 framebuffer (you'll notice because of the nicer font) instead of the earlier EFI one.

I guess amdgpu might work in case of i915 if you've got an AMD CPU with built-in GPU. This seems to be documented on the OVMF Wiki page https://wiki.archlinux.org/title/PCI_pa … -pci_early :

Note: If you also have another driver loaded this way for early modesetting (such as nouveau, radeon, amdgpu, i915, etc.), all of the aforementioned VFIO modules must precede it.

djmwj1

Member

Registered: 2013-10-04

Posts: 30

Re: Boot stucks when adding vfio-pci IDs

I guess amdgpu might work in case of i915 if you've got an AMD CPU with built-in GPU. This seems to be documented on the OVMF Wiki page https://wiki.archlinux.org/title/PCI_pa … -pci_early :

Just tried this on my system. I have two amd cards and passthrough one with the vfio-pci.ids= kernel parameter and it works. I am now able to get the prompt to decrypt my drives on boot.

To be clear, my working mkinitcpio.conf modules line is:

MODULES=(nls_cp437 crc32c kvmfr vendor-reset amdgpu vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

unnilquadium

Member

Registered: 2017-08-27

Posts: 68

Re: Boot stucks when adding vfio-pci IDs

I have the same problem. Any kernel before version 6.0 works normally.

Has anyone submitted a bug report on this?

Last edited by unnilquadium (2022-11-01 19:38:49)

luiscrjunior

Member

Registered: 2020-10-24

Posts: 2

Re: Boot stucks when adding vfio-pci IDs

Same problem here.

I have two NVIDIA GPU`s, so adding amdgu to modules array didn`t fix.

Solved (temporarily) changing kernel to linux-lts and nvidia driver to nvidia-lts.

Had do generate initramfs and update grub, as well.

Last edited by luiscrjunior (2023-01-21 15:22:25)

Xenograph

Member

Registered: 2015-01-22

Posts: 9

Re: Boot stucks when adding vfio-pci IDs

Also experiencing the same issue. Two GPUs, one NVIDIA 1050 GT and one Radeon RX 5700. I want vfio to only grab the Radeon but I end up with no frame buffers.

I've downgraded to linux-lts in the meantime. Is there any work on a fix?

seth

Member

Registered: 2012-09-03

Posts: 35,182

Re: Boot stucks when adding vfio-pci IDs

Did you read posts #6 & #7 ?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Your xinitrc is broken