You are not logged in.

#1 2024-05-04 17:44:31

M440
Member
Registered: 2023-08-24
Posts: 28

[solved] single GPU passthrouth - cant unbind console

hello,

im trying to perform single GPU passthrough to VM,

from guide:
https://github.com/ilayna/Single-GPU-pa … n/start.sh

the preperation is running following script (unbinding from consoles, unbinding from efi framebuffer, removing modules, unbinding gpu from driver)

unfortunatelly this does not work because IMO the console keeps rebinding itself or I dont understand something. Please help

#!/bin/bash
# Helpful to read output when debugging
set -x

## Load the config file with our environmental variables
source "/etc/libvirt/hooks/kvm.conf"

# Stop display manager (KDE specific)
systemctl stop display-manager

# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI-Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# Avoid a race condition
sleep 5

# Unload all Nvidia drivers
modprobe -r nvidia_drm
modprobe -r nvidia_uvm
modprobe -r nvidia_modeset
modprobe -r drm_kms_helper
modprobe -r nvidia
modprobe -r i2c_nvidia_gpu
modprobe -r drm

# Unbind the GPU from display driver
virsh nodedev-detach $VIRSH_GPU_VIDEO
virsh nodedev-detach $VIRSH_GPU_AUDIO

# Load VFIO kernel module
modprobe vfio
modprobe vfio_pci
modprobe vfio_iommu_type1

the script after my debuging modification:

#!/bin/bash

LOG=/home/mm/qemu_hooks_log.txt
VIRSH_GPU_VIDEO=pci_0000_01_00_0
VIRSH_GPU_AUDIO=pci_0000_01_00_1

# Helpful to read output when debugging
set -x

echo "Consoles at start:" >> "$LOG"
cat /sys/class/vtconsole/vtcon0/bind >> "$LOG"
cat /sys/class/vtconsole/vtcon1/bind >> "$LOG"

# Stop display manager (KDE specific)
systemctl stop display-manager

sleep 3

# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
sleep 3

echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

sleep 5

echo Sleep5Done >> "$LOG"

systemctl status display-manager >> "$LOG"
cat /proc/iomem >> "$LOG"
echo "Consoles after:" >> "$LOG"
cat /sys/class/vtconsole/vtcon0/bind >> "$LOG"
cat /sys/class/vtconsole/vtcon1/bind >> "$LOG"

modprobe -r nvidia_uvm >> "$LOG" 2>&1
sleep 1
modprobe -r nvidia_drm >> "$LOG" 2>&1
sleep 1
modprobe -r nvidia_modeset >> "$LOG" 2>&1
sleep 1
modprobe -r drm_kms_helper >> "$LOG" 2>&1
sleep 1
modprobe -r nvidia >> "$LOG" 2>&1
sleep 1
modprobe -r i2c_nvidia_gpu >> "$LOG" 2>&1
sleep 1
modprobe -r drm >> "$LOG" 2>&1

lsmod | rg nvidia >> qemu_hooks_log.txt

sleep 3

echo "virsh nodedev:" >> "$LOG"
virsh nodedev-detach "$VIRSH_GPU_VIDEO" >> "$LOG" 2>&1
virsh nodedev-detach "$VIRSH_GPU_AUDIO" >> "$LOG" 2>&1

sleep 10

echo "rebooting" >> "$LOG"
reboot

output:

Consoles at start:
0
1
EFIbufferDone
Sleep5Done
○ sddm.service - Simple Desktop Display Manager
     Loaded: loaded (/usr/lib/systemd/system/sddm.service; enabled; preset: disabled)
     Active: inactive (dead) since Sat 2024-05-04 19:23:24 CEST; 11s ago
   Duration: 1min 7.486s
       Docs: man:sddm(1)
             man:sddm.conf(5)
    Process: 757 ExecStart=/usr/bin/sddm (code=exited, status=0/SUCCESS)
   Main PID: 757 (code=exited, status=0/SUCCESS)
        CPU: 936ms

May 04 19:22:18 desktop sddm-helper[939]: Starting Wayland user session: "/usr/share/sddm/scripts/wayland-session" "/usr/lib/plasma-dbus-run-session-if-needed /usr/bin/startplasma-wayland"
May 04 19:22:18 desktop sddm[757]: Session started true
May 04 19:23:24 desktop sddm[757]: Signal received: SIGTERM
May 04 19:23:24 desktop systemd[1]: Stopping Simple Desktop Display Manager...
May 04 19:23:24 desktop sddm[757]: Auth: sddm-helper exited with 255
May 04 19:23:24 desktop sddm[757]: Display server stopping...
May 04 19:23:24 desktop sddm[757]: Display server stopped.
May 04 19:23:24 desktop sddm[757]: Running display stop script  QList("/usr/share/sddm/scripts/Xstop")
May 04 19:23:24 desktop systemd[1]: sddm.service: Deactivated successfully.
May 04 19:23:24 desktop systemd[1]: Stopped Simple Desktop Display Manager.
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000a0000-000fffff : Reserved
  000a0000-000dffff : PCI Bus 0000:00
  000f0000-000fffff : System ROM
00100000-09afefff : System RAM
09aff000-09ffffff : Reserved
0a000000-0a1fffff : System RAM
0a200000-0a20ffff : ACPI Non-volatile Storage
0a210000-0affffff : System RAM
0b000000-0b020fff : Reserved
0b021000-7fc95017 : System RAM
7fc95018-7fcb9657 : System RAM
7fcb9658-7fcef017 : System RAM
7fcef018-7fcf9e57 : System RAM
7fcf9e58-820f3fff : System RAM
820f4000-820f4fff : Reserved
820f5000-8727efff : System RAM
8727f000-8d27efff : Reserved
  8d25e000-8d261fff : MSFT0101:00
    8d25e000-8d261fff : MSFT0101:00
  8d262000-8d265fff : MSFT0101:00
    8d262000-8d265fff : MSFT0101:00
8d27f000-8f27efff : ACPI Tables
8f27f000-9127efff : ACPI Non-volatile Storage
9127f000-993fefff : Reserved
993ff000-9bff8fff : System RAM
9bff9000-9bffdfff : Reserved
9bffe000-9bffffff : System RAM
9c000000-9cffffff : Reserved
9d7f3000-9fffffff : Reserved
a0000000-f6ffffff : PCI Bus 0000:00
  e0000000-efffffff : PCI ECAM 0000 [bus 00-ff]
    e0000000-efffffff : pnp 00:00
  f5000000-f60fffff : PCI Bus 0000:01
    f5000000-f5ffffff : 0000:01:00.0
      f5000000-f5ffffff : nvidia
    f6000000-f607ffff : 0000:01:00.0
    f6080000-f6083fff : 0000:01:00.1
      f6080000-f6083fff : ICH HD audio
  f6200000-f66fffff : PCI Bus 0000:03
    f6200000-f66fffff : PCI Bus 0000:04
      f6200000-f63fffff : PCI Bus 0000:07
        f6200000-f62fffff : 0000:07:00.0
          f6200000-f62fffff : 0000:07:00.0
        f6300000-f6307fff : 0000:07:00.0
      f6400000-f64fffff : PCI Bus 0000:0e
        f6400000-f647ffff : 0000:0e:00.0
        f6480000-f64803ff : 0000:0e:00.0
          f6480000-f64803ff : ahci
      f6500000-f65fffff : PCI Bus 0000:0d
        f6500000-f6507fff : 0000:0d:00.0
          f6500000-f6507fff : xhci-hcd
      f6600000-f66fffff : PCI Bus 0000:08
        f6600000-f660ffff : 0000:08:00.0
          f6600000-f660ffff : r8169
        f6610000-f6613fff : 0000:08:00.0
  f6700000-f6afffff : PCI Bus 0000:10
    f6700000-f67fffff : 0000:10:00.4
      f6700000-f67fffff : xhci-hcd
    f6800000-f68fffff : 0000:10:00.3
      f6800000-f68fffff : xhci-hcd
    f6900000-f69fffff : 0000:10:00.2
      f6900000-f69fffff : ccp
    f6a00000-f6a07fff : 0000:10:00.6
      f6a00000-f6a07fff : ICH HD audio
    f6a08000-f6a09fff : 0000:10:00.2
      f6a08000-f6a09fff : ccp
  f6b00000-f6bfffff : PCI Bus 0000:11
    f6b00000-f6bfffff : 0000:11:00.0
      f6b00000-f6bfffff : xhci-hcd
  f6c00000-f6cfffff : PCI Bus 0000:0f
    f6c00000-f6c03fff : 0000:0f:00.0
      f6c00000-f6c03fff : nvme
  f6d00000-f6dfffff : PCI Bus 0000:02
    f6d00000-f6d03fff : 0000:02:00.0
      f6d00000-f6d03fff : nvme
f7210510-f721053f : MSFT0101:00
f7700000-f777ffff : amd_iommu
feb00000-feb00007 : SB800 TCO
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec10000-fec10fff : pnp 00:04
fec40000-fec40fff : AMDIF031:00
  fec40000-fec40fff : AMDIF031:00 AMDIF031:00
fed00000-fed003ff : HPET 0
  fed00000-fed003ff : PNP0103:00
fed81500-fed818ff : AMDI0030:00
  fed81500-fed818ff : AMDI0030:00 AMDI0030:00
fedc0000-fedc0fff : pnp 00:04
fee00000-fee00fff : Reserved
  fee00000-fee00fff : pnp 00:04
ff000000-ffffffff : pnp 00:04
100000000-85defffff : System RAM
  39e800000-39fbfffff : Kernel code
  39fc00000-3a0883fff : Kernel rodata
  3a0a00000-3a0c171bf : Kernel data
  3a14b4000-3a17fffff : Kernel bss
85df00000-85ef3ffff : RAM buffer
85ef40000-85fffffff : Reserved
860000000-ffffffffff : Reserved
  860000000-fdffffffff : PCI Bus 0000:00
    fa00000000-fc01ffffff : PCI Bus 0000:01
      fa00000000-fbffffffff : 0000:01:00.0
      fc00000000-fc01ffffff : 0000:01:00.0
Consoles after:
1
0
modprobe: FATAL: Module drm_kms_helper is builtin.
modprobe: FATAL: Module drm is builtin.
virsh nodedev:
Device pci_0000_01_00_0 detached

Device pci_0000_01_00_1 detached

rebooting

it is - IMO - succesfully
a) stopping sddm
b) unbinding eft-framebuffer
c) removing GPU modules
d) virsh deataching the driver

it fails to unbind both consoles - it just switches the one binded

i have screen signal the whole time until reboot

i tried unbinding the other console again in the script, but it does not work

EDIT:
nvm,
the screen is being taken over after running the vfio modules and running the vm,,
everything works fine

EDIT #2:
actually the fix on my system was extending the sleep time, which i did while debugging

# Avoid a race condition
sleep 11

Last edited by M440 (2024-05-04 20:14:38)

Offline

Board footer

Powered by FluxBB