No video output when VFIO_PCI module loaded early in multi-GPU setup

McRex007 · 2023-08-27 22:32:22

I've recently been possessed by the dumb idea of having a machine that runs 2 VMs so I can play any game I want splitscreen with another person. Two weeks in, and I've hit a bit of a wall. The setup is 2 GTX 1050s and a GT 710. VFIO PCI IDs are set in grub as a kernel parameter, VFIO_PCI, VFIO, and VFIO_IOMMU_TYPE1 are loaded as modules in mkinitcpio... and when rebooting, the display freezes. The system does work though, as evidenced by the fact I can blindly log in and reboot the system. I was able to extract the output of journalctl -b and dmesg after some tedium, but I don't know what I should be looking for, and despite my best efforts, its too much information in both to go through. I've tried looking it up too, but none of what I've found has helped.

The two outputs combined are just under 4000 lines long, and I've tried uploading them to some of the pastebin services on the wiki, but it seems all I have are ones that only host the files for 24 hours, and Hastebin just fails to save it. Are there any alternatives that won't involve another 2 weeks of trial and error just to figure out how to upload the log file?

seth · 2023-08-28 06:11:07

http://0x0.st
http://ix.io

McRex007 · 2023-08-28 10:35:20

Well, that was a lot easier than the big wall of small text made it seem last night.

http://0x0.st/H9dz.txt

seth · 2023-08-28 10:41:34

That has vfio take both devices from the get-go, most likely because of vfio-pci.ids=10de:1c81 ?

McRex007 · 2023-08-28 12:20:07

Yep, both GTX 1050s share the same PCI IDs. Thats fine though, because I do want both of them isolated for the virtual machines.

seth · 2023-08-28 12:25:44

Wahh… wrong VFIO thread

You seem to have no nvidia kernel modules for vmlinuz-linux-zen but nvidia-utils blocks nouveau.
https://archlinux.org/packages/extra/x8 … idia-dkms/

dkms status

McRex007 · 2023-08-28 13:54:19

Can I have nvidia-dkms and nvidia-470xx-dkms installed at the same time? It seems like I need nvidia-470xx-dkms for the GT 710.

seth · 2023-08-28 14:07:58

No, you just need some module that supports all GPUs you do not intend to forward w/ VFIO

McRex007 · 2023-08-28 14:32:02

Would the easiest way around that be to get rid of nvidia-470xx-dkms and just use Nouveau?

seth · 2023-08-28 14:44:25

You're passing through the Pascal chips, the Kepler chip is still there.
if nvidia-470xx-dkms was properly installed right now (forgot to install the kernel headers) it'd load and operate that GPU and the 470xx drivers also support the Pascal chip.

seth wrote:

dkms status

McRex007 · 2023-08-28 15:24:42

Last night when I tried to install nvidia-470xx-dkms using yay, it kept remaking the packaging and having pacman install the package over and over. I mention this because it only just now occured to me that maybe something was going wrong with installing the package, since dkms status provided no output, and then pacman -Q | grep dkms said the only package installed was the dkms package itself. Linux zen headers was installed, though. I installed nvidia-470xx-dkms, made sure it was installed this time using dkms status, and then rebooted the system... and its still happening. Heres the output of dkms status, though.

nvidia/470.199.02, 6.4.12-zen1-1-zen, x86_64: installed

seth · 2023-08-28 15:36:58

Please post an updated journal.

McRex007 · 2023-08-28 15:46:13

http://0x0.st/H95O.txt

seth · 2023-08-28 18:27:41

The nvidia kernel modules now load, so that's progress.

What most likely (hopefully so I can rant about it) get you is

Aug 28 15:39:54 archlinux kernel: [drm] Initialized simpledrm 1.0.0 20200625 for simple-framebuffer.0 on minor 0
Aug 28 15:39:54 archlinux kernel: simple-framebuffer simple-framebuffer.0: [drm] fb0: simpledrmdrmfb frame buffer device
Aug 28 15:39:54 archlinux kernel: [drm] Initialized simpledrm 1.0.0 20200625 for simple-framebuffer.0 on minor 0
Aug 28 15:39:54 archlinux kernel: simple-framebuffer simple-framebuffer.0: [drm] fb0: simpledrmdrmfb frame buffer device

because this little piece of useless shit gets to the GPU before the vfio module. It also generally just causes trouble.
Add

nvidia-drm.modeset=1

to the kernel parameters.

McRex007 · 2023-08-28 22:54:37

The funny thing is, I tried adding nvidia-drm.modeset=1 to try and get it to work, a few days ago. Fate is cruel like that, I suppose.

Edit: nvidia-drm.modeset=1 didn't fix it, though. http://0x0.st/H9Fu.txt

Last edited by McRex007 (2023-08-28 23:08:11)

seth · 2023-08-29 05:52:11

Just to be sure:
1. you've a monitor attached to the GPU you're not passing through?
2. The passed through GPUs are sufficiently isolated? https://wiki.archlinux.org/title/PCI_pa … _are_valid

McRex007 · 2023-08-29 15:19:38

Ran the script while the system was doing the video stuck thing just to be sure, and it looks like the GPUs are indeed sufficiently isolated. http://0x0.st/HpiB.txt

seth · 2023-08-29 15:50:08

Perfectly isolated…
What id you explicitly add the audio device?

vfio-pci.ids=10de:1c81,10de:0fb9

McRex007 · 2023-08-29 16:12:47

I didn't add the audio devices because the wiki said those could be detached fine with no issues. I'll try adding them and see if it makes any difference.

Edit: Nope. http://0x0.st/Hp-K.txt Added the audio device pci id, still doing the thing. Heres the journalctl output.

Last edited by McRex007 (2023-08-29 16:31:22)

seth · 2023-08-29 20:42:55

https://raw.githubusercontent.com/torva … /fbcon.rst

fbcon=map:012

Then see whether you get any output on any VT (ctrl+alt+f1,f2,f3,…)

Also, I know it sounds dumb, but

seth wrote:

1. you've a monitor attached to the GPU you're not passing through?

McRex007 · 2023-08-29 22:09:07

You'd think it sounds dumb, but then theres me, who has seen those words at least 5 times now, and still hasn't answered the question! It is attached to the GT 710, although it is through VGA, if that matters any. I'll add that to the grub config and see how it goes.

Edit: Tested with two keyboards, since my usual one is a laptop-y wireless keyboard that requires the fn key to be pressed as well to gain access to the F-keys and I was worried that might be fucking it up. Nothing with ctrl alt F1, F2, or F3.

Last edited by McRex007 (2023-08-29 22:12:09)

McRex007 · 2023-08-31 16:01:03

I haven't had too much free time on my hands, or mental fortitude, for that matter, but I have still been chipping away at it, in the event anyone else ever comes across this thread in the future. I did not have this issue with a HD 5450, as would probably be expected. I thought to check the dmesg output, but nothing stood out as relevant to the issue at hand. This time around I didn't bother getting another journalctl, but I did grab the dmesg output if anyone has a few minutes and spare eyes to offer a potential second opinion. http://0x0.st/HpMq.txt

Unfortunately at this point it seems the only viable solution would be to drop €40 on a secondhand R7 240 or some other low profile single slot GPU of the like. I can't think of anything else I could try.

Edit: Tried moving the GPU with a display attached up to the top PCIe slot, no change. WIth it still there I tried removing one, then the other, and then both of the isolated GPUs. The GT 710 only continued to display video when both of the GPUs were removed.

Last edited by McRex007 (2023-09-01 15:26:11)

Arch Linux

#1 2023-08-27 22:32:22

No video output when VFIO_PCI module loaded early in multi-GPU setup

#2 2023-08-28 06:11:07

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#3 2023-08-28 10:35:20

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#4 2023-08-28 10:41:34

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#5 2023-08-28 12:20:07

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#6 2023-08-28 12:25:44

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#7 2023-08-28 13:54:19

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#8 2023-08-28 14:07:58

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#9 2023-08-28 14:32:02

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#10 2023-08-28 14:44:25

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#11 2023-08-28 15:24:42

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#12 2023-08-28 15:36:58

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#13 2023-08-28 15:46:13

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#14 2023-08-28 18:27:41

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#15 2023-08-28 22:54:37

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#16 2023-08-29 05:52:11

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#17 2023-08-29 15:19:38

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#18 2023-08-29 15:50:08

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#19 2023-08-29 16:12:47

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#20 2023-08-29 20:42:55

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#21 2023-08-29 22:09:07

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

#22 2023-08-31 16:01:03

Re: No video output when VFIO_PCI module loaded early in multi-GPU setup

Board footer