Boot stucks when adding vfio-pci IDs

Simulacrum · 2022-10-16 11:27:32

Hey everyone

Im on a fresh Arch installation and followed the steps described in PCI passthrough via OVMF. Sadly, when i add die VFIO PCI IDs, the system doesnt boot anymore (every other step from the instruction seem to work).

The last thing my monitor shows when trying to boot is:

:: running early hook [udev]
Starting version  251.6-1-arch

Quick facts:

- The device is a NVIDIA GeForce GTX 1070
- I checked journalctl which has not a record for that boot attempt (which is not really suprinsingly).
- Removing the IDs from the kernel parameters and regenerating the GRUB config fixes the problem. Also, GRUB in recovery mode boots (but loads a Nouveau driver to my pci device)
- I'm currently running

 uname -a
Linux machine 6.0.1-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 13 Oct 2022 18:58:49 +0000 x86_64 GNU/Linux

snippets from the /etc/mkinitcpio.conf

...
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
...
HOOKS=(base udev autodetect modconf block encrypt filesystems keyboard fsck)
...

snippets from my /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX="cryptdevice=/dev/sdb2:cryptroot"
...
# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=false

I had it working on my installation before, doing the exact same things as this time (as far as i can recall it). If you need any further information, I'd be happy to provide it.

Best regards and thanks in advance,
S.

Simulacrum · 2022-10-16 15:09:11

Quick Update:

I fixed it by downgrading the packages to the 01.10.2022 by using the following Server in /etc/pacman.d/mirrorlist:

Server=https://archive.archlinux.org/repos/2022/10/01/$repo/os/$arch

So its probably a bug in one of the packages. Sadly, I do not have the time to dive deeper in the investigation.

TieMajor · 2022-10-17 06:26:57

I had the same issue expect my system was actually booting. It seems only the framebuffer was messed up (I am not using the quiet option). I am using linux-vfio as my kernel and updating to 6.0.2 is was caused the issue for me. I looked at older kernel logs to see if there was any difference but nothing jumped to me.

tsipizic · 2022-10-18 11:59:44

I have the same setup and same issue preventing boot.

Edit: Funny thing. Boot is not broken but if you have full disk encryption the password prompt is not visible. If you blindly type the password boot works and wayland starts normally
https://forum.level1techs.com/t/linux-k … ugh/190039

Last edited by tsipizic (2022-10-18 12:42:23)

Murray7501 · 2022-10-20 03:06:16

What I do is :
Don't load 'vfio-pci', 'nouveau' , 'nvidia*' kernel modules on boot.
Use a small script I found on the web, to get the 'ids=' values for vfio-pci.
-- script to get ids values --
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
--
Run the below script as root.
-- script that loads kernel modules & starts lxdm --
#!/bin/sh
# change ids= to match the passthrough GPU
modprobe vfio-pci ids=xxxx:yyyy,xxxx:yyyy
modprobe nvidia
modprobe nvidia-uvm
modprobe nvidia-drm
systemctl start lxdm & exit
--
!! Downside you lose output from the normal console until you reboot when vfio-pci is loaded !!
If someone knows a way to disable the framebuffer in UEFI-GRUB and it works around this let me know.
Hope this helps someone until fixed --

-- edit : I've looked at this a bit closer and for me its the efi-framebuffer that dies
as evident by looking in /sys/bus/platform/drivers/efi-framebuffer directory after starting up X and
then use a terminal emulator like urxvt, xterm etc. to have a look in that directory.

Last edited by Murray7501 (2022-10-21 07:22:51)

SteamedHams · 2022-10-22 14:29:47

I found a fix (or maybe, workaround) for this in my case where I use the Intel iGPU as primary CPU -- the problem seems to be that vfio-pci kills the EFI framebuffer indeed, and boot works after you blindly enter the password as once root is mounted i915 module is loaded.

You can solve this by making i915 module be loaded before vfio-pci on mkinitcpio.conf:

MODULES=(i915 vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

This makes the early password prompt be visible using the i915 framebuffer (you'll notice because of the nicer font) instead of the earlier EFI one.

I guess amdgpu might work in case of i915 if you've got an AMD CPU with built-in GPU. This seems to be documented on the OVMF Wiki page https://wiki.archlinux.org/title/PCI_pa … -pci_early :

Note: If you also have another driver loaded this way for early modesetting (such as nouveau, radeon, amdgpu, i915, etc.), all of the aforementioned VFIO modules must precede it.

djmwj1 · 2022-10-28 17:57:18

SteamedHams wrote:

I guess amdgpu might work in case of i915 if you've got an AMD CPU with built-in GPU. This seems to be documented on the OVMF Wiki page https://wiki.archlinux.org/title/PCI_pa … -pci_early :

Just tried this on my system. I have two amd cards and passthrough one with the vfio-pci.ids= kernel parameter and it works. I am now able to get the prompt to decrypt my drives on boot.

To be clear, my working mkinitcpio.conf modules line is:

MODULES=(nls_cp437 crc32c kvmfr vendor-reset amdgpu vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

unnilquadium · 2022-11-01 16:34:54

I have the same problem. Any kernel before version 6.0 works normally.

Has anyone submitted a bug report on this?

Last edited by unnilquadium (2022-11-01 19:38:49)

luiscrjunior · 2023-01-21 15:19:00

Same problem here.

I have two NVIDIA GPU`s, so adding amdgu to modules array didn`t fix.

Solved (temporarily) changing kernel to linux-lts and nvidia driver to nvidia-lts.

Had do generate initramfs and update grub, as well.

Last edited by luiscrjunior (2023-01-21 15:22:25)

Xenograph · 2023-01-26 06:12:25

Also experiencing the same issue. Two GPUs, one NVIDIA 1050 GT and one Radeon RX 5700. I want vfio to only grab the Radeon but I end up with no frame buffers.

I've downgraded to linux-lts in the meantime. Is there any work on a fix?

seth · 2023-01-26 09:08:53

Did you read posts #6 & #7 ?

luiscrjunior · 2023-02-21 00:39:28

linux-lts refers now to a 6+ kernel version.

Had to downgrade kernel manually using pacman cache:

# cd /var/cache/pacman/pkg
# pacman -U linux-lts-5.15.94-1-x86_64.pkg.tar.zst
# pacman -U nvidia-lts-1:525.89.02-3-x86_64.pkg.tar.zst
# pacman -U linux-lts-headers-5.15.94-1-x86_64.pkg.tar.zst

Also, added a IgnorePkg entry to /etc/pacman.conf file:

IgnorePkg   = linux linux-headers linux-lts linux-lts-headers nvidia nvidia-lts

So, pacman will ignore further updates:

 -> linux-lts: ignoring package upgrade (5.15.94-1 => 6.1.12-1)
 -> linux-lts-headers: ignoring package upgrade (5.15.94-1 => 6.1.12-1)
 -> nvidia-lts: ignoring package upgrade (1:525.89.02-3 => 1:525.89.02-4)

seth · 2023-02-21 07:17:51

Please don't PSA the wiki, https://wiki.archlinux.org/title/Pacman … g_upgraded
Also:

seth wrote:

Did you read posts #6 & #7 ?

luiscrjunior · 2023-02-21 14:45:47

seth wrote:

Did you read posts #6 & #7 ?

Yes, no difference. My setup has two dedicated NVIDIA (one for host and other for qemu) and no integrated graphics card. No difference adding amdgu or i915. Only thing that solved was downgrading kernel to a < 6 version. I'm stuck. But thanks.

seth · 2023-02-21 15:18:52

No difference adding amdgu or i915

Quiter obviously not. There're however wiki links that point out

If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.

and how to do that.
Are you?

luiscrjunior · 2023-02-21 17:05:54

seth wrote:

No difference adding amdgu or i915
Quiter obviously not. There're however wiki links that point out
If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.
and how to do that.
Are you?

Also tried. No success.

seth · 2023-02-21 17:39:58

Are you sure you "tried" correctly?

luiscrjunior · 2023-02-21 18:46:24

seth wrote:

Are you sure you "tried" correctly?

Sure.

Updating:

I found a comment in reddit that reverting this commit solved.

I tried here and worked.

Just to test, I reverse patched the c file, compiled kernel, (plus regenerated initramfs, updated bootloader, reinstalled nvidia drivers) and now I have boot messages again with 6.2 kernel and vfio (GPU passthrough) working flawlessly.

Last edited by luiscrjunior (2023-02-21 18:48:34)

defekkt · 2023-03-17 16:55:30

same issue here, was on lts as a workaround until it got bumped to 6.x as well...

not sure I'd call a recompile with a reverted commit a solution though :-S

found a solution using a initramfs script:
https://hardcoded.info/post/2023/03/04/ … -solution/

Last edited by defekkt (2023-03-21 15:43:19)

luiscrjunior · 2023-04-05 11:47:50

defekkt wrote:

same issue here, was on lts as a workaround until it got bumped to 6.x as well...
not sure I'd call a recompile with a reverted commit a solution though :-S
found a solution using a initramfs script:
https://hardcoded.info/post/2023/03/04/ … -solution/

Thanks for the link! Worked here!

Arch Linux

#1 2022-10-16 11:27:32

Boot stucks when adding vfio-pci IDs

#2 2022-10-16 15:09:11

Re: Boot stucks when adding vfio-pci IDs

#3 2022-10-17 06:26:57

Re: Boot stucks when adding vfio-pci IDs

#4 2022-10-18 11:59:44

Re: Boot stucks when adding vfio-pci IDs

#5 2022-10-20 03:06:16

Re: Boot stucks when adding vfio-pci IDs

#6 2022-10-22 14:29:47

Re: Boot stucks when adding vfio-pci IDs

#7 2022-10-28 17:57:18

Re: Boot stucks when adding vfio-pci IDs

#8 2022-11-01 16:34:54

Re: Boot stucks when adding vfio-pci IDs

#9 2023-01-21 15:19:00

Re: Boot stucks when adding vfio-pci IDs

#10 2023-01-26 06:12:25

Re: Boot stucks when adding vfio-pci IDs

#11 2023-01-26 09:08:53

Re: Boot stucks when adding vfio-pci IDs

#12 2023-02-21 00:39:28

Re: Boot stucks when adding vfio-pci IDs

#13 2023-02-21 07:17:51

Re: Boot stucks when adding vfio-pci IDs

#14 2023-02-21 14:45:47

Re: Boot stucks when adding vfio-pci IDs

#15 2023-02-21 15:18:52

Re: Boot stucks when adding vfio-pci IDs

#16 2023-02-21 17:05:54

Re: Boot stucks when adding vfio-pci IDs

#17 2023-02-21 17:39:58

Re: Boot stucks when adding vfio-pci IDs

#18 2023-02-21 18:46:24

Re: Boot stucks when adding vfio-pci IDs

#19 2023-03-17 16:55:30

Re: Boot stucks when adding vfio-pci IDs

#20 2023-04-05 11:47:50

Re: Boot stucks when adding vfio-pci IDs

Board footer