[SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

faceyneck · 2019-12-22 00:24:22

EDIT3: After some farting around, and doing a LOT of reading and experimenting, I have actually successfully configured GPU passthrough to a kernel-based virtual machine. The solution was to disable the descrete GPU in UEFI, and keep trying things over and over again. I was also trying a recommendation from a post about iommu on github for Arch, but for whatever reason, the mkinitcpio flag he used was unusual to me. Sorry, don't remember what it was. But once I finally ran mkinitcpio with the normal flag (normal for me anyway; mkinitcpio -p <kernel1><kernel2><kernel3>...) I was not only able to get it working, but also able to get it working without the vfio kernel from the AUR. I think this is enough explanation for this post. I plan on making a detailed account of everything once I'm done tinkering with it. At the moment, I do not have keyboard/mouse passthrough configured, and I do not have all drivers setup either. Also, if you're a new user at some future time reading this to troubleshoot iommu, and this explanation isn't enough information, iommu is probably outside the scope of your Linux proficiency. Sorry. Guess you should get on that Arck Wiki, find fun things to implement, break things and then start over. You'll be a Superuser in no time. :-)

EDIT2: I obtained a second graphics card, so this issue is no longer relevant to me.

EDIT1: Forgot to mention: System is encrypted lvm and luks1. And I'm attempting this with Intel integrated graphics. I can get another graphics card, but from what I've read it doesn't seem to be necessary?
I'm positive my card and CPU support IOMMU/Vt-d, and are enabled in UEFI.
Obligatory system information:

 

                   -`                    rj@archlinux 
                  .o+`                   ------------ 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Host: All Series 
               `+oooooo:                 Kernel: 5.4.5-zen1-1-zen 
               -+oooooo+:                Uptime: 11 mins 
             `/:-:++oooo+:               Packages: 1536 (pacman) 
            `/++++/+++++++:              Shell: bash 5.0.11 
           `/++++++++++++++:             Resolution: 2560x1440, 1366x768 
          `/+++ooooooooooooo/`           DE: Plasma 
         ./ooosssso++osssssso+`          WM: KWin 
        .oossssso-````/ossssss+`         Theme: Breeze-Dark [GTK2/3] 
       -osssssso.      :ssssssso.        Icons: breeze-dark [GTK2/3] 
      :osssssss/        osssso+++.       Terminal: yakuake 
     /ossssssss/        +ssssooo/-       CPU: Intel i7-4790K (8) @ 4.400GHz 
   `/ossssso+/:-        -:/+osssso+-     GPU: AMD ATI Radeon R9 290X/390X 
  `+sso+:-`                 `.-/+oso:    GPU: Intel HD Graphics 
 `++:.                           `-/+/   Memory: 2096MiB / 15949MiB 
 .`                                 `/

IMPORTANT EDIT:

After running the script mentioned in the wiki Ensure Groups are Valid, it looks like I can NOT use integrated graphics, as my CPU and GPU are in the same group. Here's the relevant output from the script:

 

IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]

...so does this mean I need another graphics card/cannot passthrough my graphics card, and have integrated graphics for host/Arch LInux?

I wasn't sure where to post this. It could be considered a n00b issue; it could be considered an advanced one.

The short story is this; I make the kernel parameter file

/etc/modprobe.d/vfio.conf

as so:

  


options vfio-pci ids=1002:67b0,1002:aac8

I'm sure those are the appropriate hardware ID's. Gathered from

 lspci -nn

Here's the MODULES line from

 /etc/mkinitcpio.conf

:

 MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd amdgpu radeon"

...and that's where I'm at now. If I rebuild initramfs, the system won't boot. It will start to load the "early kernel modules" or whatever, and hang after "LVM2" IIRC.

So then I need to boot from LiveCD, remove the kernel modules from mkinitcpio.conf, regenerate initramfs, and reboot into my system.

I'm not sure where to check logs to get more information than that.

I'm still configuring things, but I'm gonna go ahead and mark this as [SOLVED]

Last edited by faceyneck (2020-04-23 01:46:15)

Lone_Wolf · 2019-12-22 12:49:59

IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]

That group consists of 3 devices (radeon video, radeon audio and an SSD) with 2 PCIe controllers.
Your CPU and iGPU are not in that group.

Post the full output of the script mentioned.

options vfio-pci ids=1002:67b0,1002:aac8

Those IDs are from a Hawaii XT / Grenada XT [Radeon R9 290X/390X] videocard and Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] .

Do you want to passthrough the amd gpu and have linux/host use the intel integrated one ?

Edit : removed further instructions until it's clear what OP wants.

Last edited by Lone_Wolf (2019-12-22 12:52:14)

faceyneck · 2019-12-26 01:55:52

Here's the full output of the script from the wiki:

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
        00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 10:
        00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU Group 11:
        00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)
IOMMU Group 12:
        00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0)
IOMMU Group 13:
        00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU Group 14:
        00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4]
        00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
        00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU Group 15:
        04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 16:
        05:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 17:
        05:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
        07:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU Group 18:
        05:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
        08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 19:
        09:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU Group 2:
        00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
IOMMU Group 20:
        0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 3:
        00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
IOMMU Group 4:
        00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 5:
        00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 6:
        00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1]
IOMMU Group 7:
        00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 8:
        00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 9:
        00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)

I want to passthrough the AMD "discrete" graphics card, and keep Arch on the integrated graphics from the 4790K processor.

I hope that's all the information you asked for. Thanks for your response.

It still looks to me like my integrated graphics and discrete graphics are grouped together.

EDIT: By that I mean; the damn AMD graphics card doesn't show up in any other group, right? I might have glanced over it. Seems to me it's only ever showing up in group 1, vis a vis, can only passthrough the card in group 1.

Last edited by faceyneck (2019-12-26 01:57:59)

V1del · 2019-12-26 11:18:04

The integrated is in group 2, and the discrete in group 1, so everything dandy here. What makes me slightly worried in that output is that the integrated one is marked as a display controller, which usually means that you don't have an output you you could plug something into. I don't know enough of passthrough to know if that will make it impossible, but it will certainly complicate things.

Lone_Wolf · 2019-12-28 19:58:07

The SSD and discrete card are in the same IOMMU group, not passing through the SSD will likely require an ACS-enabled kernel, like linux-vfio in aur.

Getting the integrated card to display may be trickier though.
Try connecting a screen to it , selecting it as "boot first" in firmware and test.

faceyneck · 2020-01-02 21:09:57

I currently have my HDMI monitor plugged into integrated graphics; DVI-D monitor plugged into discrete graphics card, AKA AMD R9 290X.

...so in regards to the Integrated graphics, they're most certainly displaying. Maybe I'm misunderstanding what you're asking?

Thanks again for the replies, folks.

EDIT: I thought I had read that the Zen kernel provided all the vfio features as the vfio kernel from AUR. Is that not the case?

Last edited by faceyneck (2020-01-02 21:12:16)

Lone_Wolf · 2020-01-03 10:53:49

There are 2 issues that need investigation :

1.
Your system has an integrated intel gpu and a dedicated amd gpu, which makes it a hybrid system.
Usually those systems allow only one card to drive the outputs, for passthrough however you need atleast 2 cards that can function independently .

2.
all supported archlinux kernels (including zen kernel) are configured for vfio support.
Without a special patch vfio only allows to passthrough an entire iommu group.
To override that behaviour a special patch is needed, for systems with intel igpu another patch is needed.
Afaik the aur linux-vfio kernel is the only kernel for archlinux that has those patches.

Let's investigate a bit further.
post the outputs of

$ xrandr --listproviders
$ glxinfo -B
$ DRI_PRIME=1 glxinfo -B

run glxgears , on which monitor does it display ?
run DRI_PRIME=1 glxgears , on which monitor do you see the rotating gears ?

faceyneck · 2020-01-31 22:59:44

Mkay, here's the requested outputs:

xrandr --listproviders
Providers: number : 2
Provider 0: id: 0x59 cap: 0xf, Source Output, Sink Output, Source Offload, Sink Offload crtcs: 6 outputs: 4 associated providers: 1 name:AMD Radeon R9 200 Series @ pci:0000:01:00.0
Provider 1: id: 0x8d cap: 0xf, Source Output, Sink Output, Source Offload, Sink Offload crtcs: 3 outputs: 5 associated providers: 1 name:modesetting

glxinfo -B
name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: X.Org (0x1002)
    Device: AMD Radeon R9 200 Series (HAWAII, DRM 3.35.0, 5.4.11-zen1-1-zen, LLVM 11.0.0) (0x67b0)
    Version: 20.0.0
    Accelerated: yes
    Video memory: 4096MB
    Unified memory: no
    Preferred profile: core (0x1)
    Max core profile version: 4.6
    Max compat profile version: 4.6
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
    VBO free memory - total: 3608 MB, largest block: 3608 MB
    VBO free aux. memory - total: 4001 MB, largest block: 4001 MB
    Texture free memory - total: 3608 MB, largest block: 3608 MB
    Texture free aux. memory - total: 4001 MB, largest block: 4001 MB
    Renderbuffer free memory - total: 3608 MB, largest block: 3608 MB
    Renderbuffer free aux. memory - total: 4001 MB, largest block: 4001 MB
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 4096 MB
    Total available memory: 8192 MB
    Currently available dedicated video memory: 3608 MB
OpenGL vendor string: X.Org
OpenGL renderer string: AMD Radeon R9 200 Series (HAWAII, DRM 3.35.0, 5.4.11-zen1-1-zen, LLVM 11.0.0)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 4.6 (Compatibility Profile) Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile

OpenGL ES profile version string: OpenGL ES 3.2 Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

 

DRI_PRIME=1 glxinfo -B
name of display: :0
libGL error: Different GPU, but blitImage not implemented for this driver
libGL error: failed to load driver: i965
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Intel Open Source Technology Center (0x8086)
    Device: Mesa DRI Intel(R) HD Graphics 4600 (HSW GT2) (0x412)
    Version: 20.0.0
    Accelerated: yes
    Video memory: 1536MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 3.0
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.1
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) HD Graphics 4600 (HSW GT2)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 3.0 Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL shading language version string: 1.30
OpenGL context flags: (none)

OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.0.0-devel (git-8405e1bef0)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

...and if I run DRI_PRIME=1 glxgears , the system locks and spits me to the display manager. Is that normal?
I was in the middle up updates... let's hope I'm not worse for wear.

Lone_Wolf · 2020-02-01 18:41:27

Your system has all the signs of using hybrid graphics, combining an intel integrated GPU with a discrete AMD gpu .

Your iGPU is a HD Graphics 4600 which points to an intel processor from the haswell family around 2013/2014 .
Unfortunately many of the intel motherboard chipsets from that era were designed with intel IGPU as primary and an nivida card as secondary GPU in mind.

Do you have early mircocode loading configured , https://wiki.archlinux.org/index.php/Microcode ?

What brand and model is your motherboard and which uefi/bios firmware version are you using t ?

Sidenote :
Just above the reply window you'll find a "subscribe to this topic" link.
Use it and you'll get a mail when somebody replies to that topic.

Last edited by Lone_Wolf (2020-02-01 18:41:53)

faceyneck · 2020-03-23 01:00:32

Lone_Wolf wrote:

Your system has all the signs of using hybrid graphics, combining an intel integrated GPU with a discrete AMD gpu .
Your iGPU is a HD Graphics 4600 which points to an intel processor from the haswell family around 2013/2014 .
Unfortunately many of the intel motherboard chipsets from that era were designed with intel IGPU as primary and an nivida card as secondary GPU in mind.

Do you have early mircocode loading configured , https://wiki.archlinux.org/index.php/Microcode ?
What brand and model is your motherboard and which uefi/bios firmware version are you using t ?

Sidenote :
Just above the reply window you'll find a "subscribe to this topic" link.
Use it and you'll get a mail when somebody replies to that topic.

Thanks for that suggestion; I'll subscribe to this thread. Sorry about the delay in responding.

CPU is Haswell. 4790k, a late-release of a Haswell design.

I have configured early microcode loading, yes.

Motherboard is an ASUS Z97-Pro. I'll look up the firmware version and edit into this post.

EDIT2: I obtained a second graphics card, so this issue is no longer relevant to me. Topic changed to [CLOSED] by me.

[EDIT] This is the information about motherboard firmware/UEFI and so forth:
BIOS version: 2702 x64
Build date: 10/27/2015
EC version: MBE0 - Z97 - 0121
ME version: 9.1.1000
PCH stepping: 00/A0
I hope that's the information that you - the lovely people of Arch - had requested. Thanks again for the help everyone[/EDIT]

Last edited by faceyneck (2020-04-11 19:53:49)

Arch Linux

#1 2019-12-22 00:24:22

[SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#2 2019-12-22 12:49:59

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#3 2019-12-26 01:55:52

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#4 2019-12-26 11:18:04

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#5 2019-12-28 19:58:07

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#6 2020-01-02 21:09:57

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#7 2020-01-03 10:53:49

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#8 2020-01-31 22:59:44

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#9 2020-02-01 18:41:27

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

#10 2020-03-23 01:00:32

Re: [SOLVED]Attempting GPU Passthrough, Stuck At Rebuilding Initramfs

Board footer