iGPU passthrough

31KM · 2024-03-05 23:05:57

Hello everyone,

I try to pass the iGPU of my Ryzen 7700 to a VM. I followed the steps from the wiki and isolated the GPU:

0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c3)
Subsystem: ASUSTeK Computer Inc. Raphael [1043:8877]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu

The iGPU is the only device in the IOMMU group.
I dumped the vbios twice:
1. cat /sys/kernel/debug/dri/0e:00.0/amdgpu_vbios
2. https://github.com/isc30/ryzen-7000-series-proxmox - vbios.c
Both led to the exact same result.

I added the vbios to the proper hostdev section in the XML like so:

<rom file="/home/myuser/dev/vbios_extract/vbios_1002_164e.bin"/>

I also added the feature/hyperv/vendor_id randomness.

I tried two things:
1. VM with latest archiso
- Installed the system properly, shutdown, added the device, got error
- For quicker testing, I pass the device during the install and just check in the live env, error happens as well
The error:

[ 4.073662] amdgpu 0000:05:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
[ 4.073688] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[ 4.073692] amdgpu 0000:05:00.0: amdgpu: VRAM: 512M 0x0000FFFFFF000000 - 0x000100001EFFFFFF (512M used)
[ 4.073694] amdgpu 0000:05:00.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
[ 4.073700] [drm] Detected VRAM RAM=512M, BAR=256M
[ 4.073701] [drm] RAM width 128bits DDR5
[ 4.073777] [drm] amdgpu: 512M of VRAM memory ready
[ 4.073778] [drm] amdgpu: 7997M of GTT memory ready.
[ 4.073788] [drm] GART: num cpu pages 262144, num gpu pages 262144
[ 4.073818] ------------[ cut here ]------------
[ 4.073819] kernel BUG at drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c:490!
[ 4.073879] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 4.073887] CPU: 1 PID: 152 Comm: (udev-worker) Not tainted 6.7.6-arch1-2 #1 36a1d137df2a95849ad6b7232a6023837991924e
[ 4.073897] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022
[ 4.073906] RIP: 0010:gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu]
[ 4.074087] Code: d0 48 89 03 5b 5d 41 5c 41 5d e9 b9 7a 6e e4 e8 b9 52 f9 ff 48 89 45 00 48 89 c6 48 b8 3f 00 00 00 00 00 ff ff 48 85 c6 74 9f <0f> 0b 48 8b 03 48 0f ba e0 36 73 0a 48 0f ba f0 36 48 89 03 eb 9f
[ 4.074104] RSP: 0018:ffffae43c09b79d0 EFLAGS: 00010206
[ 4.074112] RAX: ffff00000000003f RBX: ffffae43c09b7a00 RCX: ffffae43c09b7a00
[ 4.074121] RDX: 0000000000000000 RSI: 00fe800000400000 RDI: ffff94451fa00000
[ 4.074131] RBP: ffffae43c09b79f8 R08: 00000000000186a0 R09: 0000000000000001
[ 4.074139] R10: 0000000000001000 R11: 0000000000001000 R12: 00000000ffffffff
[ 4.074148] R13: ffff94451fa00000 R14: 0000000000000000 R15: 0000000000000001
[ 4.074158] FS: 000071dab4b90540(0000) GS:ffff94486fc80000(0000) knlGS:0000000000000000
[ 4.074168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4.074176] CR2: 000071dab559a8c0 CR3: 000000011b9a4000 CR4: 0000000000750ef0
[ 4.074186] PKRU: 55555554
[ 4.074190] Call Trace:
[ 4.074196] <TASK>
[ 4.074201] ? die+0x36/0x90
[ 4.074209] ? do_trap+0xda/0x100
[ 4.074216] ? gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.074365] ? do_error_trap+0x6a/0x90
[ 4.074371] ? gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.074501] ? exc_invalid_op+0x50/0x70
[ 4.074508] ? gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.074638] ? asm_exc_invalid_op+0x1a/0x20
[ 4.074647] ? gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.074776] amdgpu_gmc_pd_addr+0x4e/0x80 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.074907] gfxhub_v2_1_gart_enable+0x9a/0xfe0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.075037] gmc_v10_0_hw_init+0x170/0x1d0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.075163] amdgpu_device_init+0x147a/0x2470 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.075291] ? __entry_text_end+0x101f76/0x101f79
[ 4.075298] ? srso_alias_return_thunk+0x5/0xfbef5
[ 4.075304] ? srso_alias_return_thunk+0x5/0xfbef5
[ 4.075312] amdgpu_driver_load_kms+0x19/0x190 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.075440] amdgpu_pci_probe+0x165/0x4c0 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.075566] local_pci_probe+0x42/0xa0
[ 4.075575] pci_device_probe+0xc1/0x260
[ 4.075583] ? sysfs_do_create_link_sd+0x6e/0xe0
[ 4.075591] really_probe+0x19b/0x3e0
[ 4.075786] ? __pfx___driver_attach+0x10/0x10
[ 4.075955] __driver_probe_device+0x78/0x160
[ 4.076122] driver_probe_device+0x1f/0x90
[ 4.076289] __driver_attach+0xd2/0x1c0
[ 4.076456] bus_for_each_dev+0x85/0xd0
[ 4.076622] bus_add_driver+0x116/0x220
[ 4.076785] driver_register+0x59/0x100
[ 4.076948] ? __pfx_amdgpu_init+0x10/0x10 [amdgpu 2d7448c597270f7a0274f1b4ff6537e961507950]
[ 4.077229] do_one_initcall+0x58/0x320
[ 4.077396] do_init_module+0x60/0x240
[ 4.077560] init_module_from_file+0x89/0xe0
[ 4.077723] idempotent_init_module+0x120/0x2b0
[ 4.077883] __x64_sys_finit_module+0x5e/0xb0
[ 4.078039] do_syscall_64+0x61/0xe0
[ 4.078189] ? srso_alias_return_thunk+0x5/0xfbef5
[ 4.078333] ? do_syscall_64+0x70/0xe0
[ 4.078475] ? syscall_exit_to_user_mode+0x2b/0x40
[ 4.078616] ? srso_alias_return_thunk+0x5/0xfbef5
[ 4.078757] ? do_syscall_64+0x70/0xe0
[ 4.078897] entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 4.079041] RIP: 0033:0x71dab566c88d
[ 4.079186] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 73 14 0d 00 f7 d8 64 89 01 48
[ 4.079488] RSP: 002b:00007ffcaf51dcc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 4.079639] RAX: ffffffffffffffda RBX: 000064a985d0bc90 RCX: 000071dab566c88d
[ 4.079795] RDX: 0000000000000000 RSI: 000071dab5781376 RDI: 0000000000000025
[ 4.079947] RBP: 000071dab5781376 R08: 0000000000000001 R09: fffffffffffffe88
[ 4.080101] R10: 0000000000000050 R11: 0000000000000246 R12: 0000000000020000
[ 4.080256] R13: 000064a985d0c3b0 R14: 0000000000000000 R15: 000064a985d0e700
[ 4.080414] </TASK>
[ 4.080567] Modules linked in: hid_generic usbhid amdgpu(+) virtio_gpu virtio_dma_buf virtio_rng crc32_pclmul virtio_net crc32c_intel net_failover sha512_ssse3 virtio_balloon serio_raw virtio_console virtio_blk failover atkbd sha256_ssse3 libps2 amdxcp vivaldi_fmap drm_exec aesni_intel gpu_sched drm_buddy crypto_simd video sr_mod wmi i2c_algo_bit drm_suballoc_helper cryptd drm_ttm_helper ttm xhci_pci virtio_pci xhci_pci_renesas cdrom drm_display_helper cec virtio_pci_legacy_dev intel_agp i8042 virtio_pci_modern_dev intel_gtt serio
[ 4.081448] ---[ end trace 0000000000000000 ]---
[ 4.081852] RIP: 0010:gmc_v10_0_get_vm_pde+0x9d/0xc0 [amdgpu]
[ 4.082268] Code: d0 48 89 03 5b 5d 41 5c 41 5d e9 b9 7a 6e e4 e8 b9 52 f9 ff 48 89 45 00 48 89 c6 48 b8 3f 00 00 00 00 00 ff ff 48 85 c6 74 9f <0f> 0b 48 8b 03 48 0f ba e0 36 73 0a 48 0f ba f0 36 48 89 03 eb 9f
[ 4.082743] RSP: 0018:ffffae43c09b79d0 EFLAGS: 00010206
[ 4.083026] RAX: ffff00000000003f RBX: ffffae43c09b7a00 RCX: ffffae43c09b7a00
[ 4.083305] RDX: 0000000000000000 RSI: 00fe800000400000 RDI: ffff94451fa00000
[ 4.083605] RBP: ffffae43c09b79f8 R08: 00000000000186a0 R09: 0000000000000001
[ 4.083990] R10: 0000000000001000 R11: 0000000000001000 R12: 00000000ffffffff
[ 4.084297] R13: ffff94451fa00000 R14: 0000000000000000 R15: 0000000000000001
[ 4.084581] FS: 000071dab4b90540(0000) GS:ffff94486fc80000(0000) knlGS:0000000000000000
[ 4.084863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4.085153] CR2: 000071dab559a8c0 CR3: 000000011b9a4000 CR4: 0000000000750ef0
[ 4.085493] PKRU: 55555554

Full dmesg: https://termbin.com/jseh
Removing the virtual graphics and display show the bootmenu in the virt-manager window, once I select something the physical screen stays black.
Host kernel is 6.7.8-arch1, while guest is 6.7.6-arch1.

2. VM with KDE neon
Also add device including vbios and randomness before install. System boots without the previously mentioned error, but if I remove the the virtual graphics and display the bootmenu is shown in the virt-manager window, once I select something the physical screen stays black. With the virtual display I can see that the amdgpu driver is properly loaded. Kernel version is some LTS (Ubuntu 22.04 LTS).

Any ideas? If you need more info I will gladly provide.

Last edited by 31KM (2024-03-05 23:08:54)

31KM · 2024-04-17 20:16:59

I got it working with a Windows 11 guest. I also needed to pass the iGPU audio device (Rembrandt) in addition to the actual iGPU VGA:

0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8877]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

That alone was not enough, as the card failed with Code43 in Windows. According to https://www.reddit.com/r/VFIO/comments/ … ssthrough/ that is caused by the fact that the used VGA bios does not support EFI.
I used the AMDGopDriver with EFI support from here: https://github.com/isc30/ryzen-7000-ser … Driver.rom

Also the reset bug fix is required: https://oomza.cutegay.software/inga-lov … esetBugFix

Looking glass then also worked (steps from the wiki), albeit with small rendering issues (some letters missing and such).

Arch Linux

#1 2024-03-05 23:05:57

iGPU passthrough

#2 2024-04-17 20:16:59

Re: iGPU passthrough

Board footer