Arch Linux

stonecraft

Member

Registered: 2019-06-25

Posts: 14

Another VFIO binding problem with nouveau [solved]

I have a problem similar to this one: https://bbs.archlinux.org/viewtopic.php?id=245670
I have been trying to get passthrough of a GTX1070 to work with an Ubuntu Linux guest. Passthrough to Windows 10 works fine, but with Linux guests I just get crashes

IOMMU is enabled

[boss@virtland ~]$ dmesg | grep -i -e DMAR -e IOMMU
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/main--VG-lvroot rw amd_iommu=on hugepages=4096 audit=0 loglevel=3 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/main--VG-lvroot rw amd_iommu=on hugepages=4096 audit=0 loglevel=3 quiet
[    0.420885] AMD-Vi: IOMMU performance counters supported
[    0.420934] AMD-Vi: IOMMU performance counters supported
[    0.422189] iommu: Adding device 0000:00:01.0 to group 0
[    0.422848] iommu: Adding device 0000:00:01.1 to group 1
[    0.423525] iommu: Adding device 0000:00:01.2 to group 2

Card is in a separate IOMMU group from host GPU (edited to show relevant groups):

shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
IOMMU Group 38:
	42:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
IOMMU Group 39:
	43:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
	43:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 4:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 40:
	44:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
	44:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

vfio.conf is setup:

$ cat /etc/modprobe.d/vfio.conf
alias pci:v00001002d000067DFsv00001462sd00003414bc03sc00i00 vfio-pci
alias pci:v00001002d0000AAF0sv00001462sd0000AAF0bc04sc03i00 vfio-pci
options vfio-pci ids=1002:67df,1002:aaf0

#and nvidia
alias pci:v000010DEd00001B81sv00003842sd00005173bc03sc00i00 vfio-pci
alias pci:v000010DEd000010F0sv00003842sd00005173bc04sc03i00 vfio-pci
options vfio-pci ids=10de:1b81,10de:10f0

#the next two options aren't from the guide, but they don't seem to make a difference either way
options vfio-pci disable_vga=1
softdep nouveau pre:vfio-pci

#and USB card
alias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 vfio-pci
options vfio-pci ids=1b21:1242

mkinitcpio (comments removed for brevity):

# vim:set ft=sh
# MODULES
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

# BINARIES
BINARIES=()

# FILES
FILES=(/etc/modprobe.d/kvm.conf /etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)

# HOOKS
HOOKS=(base udev autodetect modconf vfio block lvm2 filesystems keyboard fsck)

and this is the relevant section of journalctl:
http://codepad.org/aOKXZPDZ

There's stuff related to nouveau there. Previously when I encountered this problem, the system crashed when I tried to unbind the nouveau driver from the GPU I am passing, but now it appears to work:

echo '0000:43:00.0' | sudo tee /sys/bus/pci/drivers/nouveau/unbind

[boss@virtland ~]$ echo '0000:43:00.0' | sudo tee /sys/bus/pci/drivers/nouveau/unbind
0000:43:00.0
Killed

But the host still crashes when I try to start the VM with the passthrough stuff in the XML file.
This is the domain XML:
http://codepad.org/IpDm0ol6

What else can I try?

Last edited by stonecraft (2019-08-09 08:49:21)

stonecraft

Member

Registered: 2019-06-25

Posts: 14

Re: Another VFIO binding problem with nouveau [solved]

Update: if I comment out the module alias entry for my USB card, the GTX1070 in /etc/modprobe.d/vfio.conf, the VM can start, but then gets stuck with a ubuntu splash screen during startup on both the VNC and the passed-through monitor. The three dots are moving, so it seems that Ubuntu isn't actually frozen. Also, I can send a shutdown command via virt-manager, also showing that the VM is not entirely frozen. And unlike before, I can restart the host without incident.
The driver is also now listed by the host as vfio-pci.

Maybe this is a problem on the guest side?

Edit: I had forgotten to add the vendorid and kvm=off stuff. So now I can get past the Ubuntu splash screen, but then the vnc client goes purple and the monitor attached to the passed-through card is displaying rainbow gibberish (but there is a working pointer).
Edit2: Removing the emulated graphics and video fixed the screen, so I get a normal login screen, but it won't actually log in. So it seems my remaining problems are guest-side.
Edit3: Installing the proprietary NVIDIA driver on the guest allowed me to log in normally

But none of that explains why module aliases would interfere with each other, which is the problem that caused me to start this thread. It seems that wasn't essential anyway as the USB 3.0 card I was aliasing seems to be properly captured by my other guest without issue.

$ cat /etc/modprobe.d/vfio.conf
alias pci:v00001002d000067DFsv00001462sd00003414bc03sc00i00 vfio-pci
alias pci:v00001002d0000AAF0sv00001462sd0000AAF0bc04sc03i00 vfio-pci
options vfio-pci ids=1002:67df,1002:aaf0

#and nvidia
alias pci:v000010DEd00001B81sv00003842sd00005173bc03sc00i00 vfio-pci
alias pci:v000010DEd000010F0sv00003842sd00005173bc04sc03i00 vfio-pci
options vfio-pci ids=10de:1b81,10de:10f0

#the next two options aren't from the guide, but they don't seem to make a difference either way
#options vfio-pci disable_vga=1
#softdep nouveau pre:vfio-pci

#and USB card
#alias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 vfio-pci
#options vfio-pci ids=1b21:1242

Last edited by stonecraft (2019-08-09 03:35:27)

stonecraft

Member

Registered: 2019-06-25

Posts: 14

Re: Another VFIO binding problem with nouveau [solved]

Should I mark this as solved? I got past the problem that started the thread, but there is still a mystery here for me (why did aliasing my USB 3.0 card interfere with the GPU passthrough?)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,300

Re: Another VFIO binding problem with nouveau [solved]

Yeah please do, FWIW as far as I know if you specify multiple option lines all adjusting the same parameter the last one will take precedence, so you'd only pass through your USB and not the nvidia card. I'm not 100% sure of this claim right now and I'm not in front of my Arch, but use

systool -vm vfio_pci

to verify. Add all id's you want to pass through to a single options invocation.

Last edited by V1del (2019-08-09 08:09:05)

stonecraft

Member

Registered: 2019-06-25

Posts: 14

Re: Another VFIO binding problem with nouveau [solved]

Thanks, I was never quite clear on how to deal with options for multiple cards. I know I had multiple lines of options in there before and everything worked ok, but the order was probably different or something.

Is this the correct way?

#AMD GPU
alias pci:v00001002d000067DFsv00001462sd00003414bc03sc00i00 vfio-pci
alias pci:v00001002d0000AAF0sv00001462sd0000AAF0bc04sc03i00 vfio-pci
options vfio-pci ids=1002:67df,1002:aaf0,1b21:1242
# NVIDIA
alias pci:v000010DEd00001B81sv00003842sd00005173bc03sc00i00 vfio-pci
alias pci:v000010DEd000010F0sv00003842sd00005173bc04sc03i00 vfio-pci
# USB card
alias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 vfio-pci

#options listing all cards:
options vfio-pci ids=10de:1b81,10de:10f0,1002:67df,1002:aaf0,1002:67df,1002:aaf0,1b21:1242