[SOLVED] X hangs when multiple GPUs are present

toljaga · 2021-10-30 14:42:08

Hi,

I'm trying to pass through my main GPU to a Windows VM so that I can play games and edit videos on it while I use Arch for the rest of my work, as per this guide. The card I would be using for the VM is an AMD RX 570 (so, GCN4 architecture) while Arch is supposed to get my old ATI X700 (R400 architecture). However, I ran into a completely separate problem before I could even start with that.

I've installed Arch, X11 and Wayland (though I prefer to use X11 since Wayland does not run well on the X700) with sddm and KDE/Plasma on top of that.

When either of my cards is in the x16 slot of my motherboard, everything works fine (no hangs, no error messages etc.). However, when I put the second card in and boot again, I encounter an issue.

As the title suggests, when the machine boots X does not start. The screen displays the following text:

Starting version 249.5-3-arch
/dev/nvme0n1p3: clean, 332535/6553600 files, 5213014/26214400 blocks

Which is just the last thing that shows up before I see the login screen anyway. (In case it is important, nvme0n1p3 is my main filesystem partition, p1 is boot and p2 swap, also there is sufficient space on p3; some ~80GB are free).

I should note that my motherboard (a GIGABYTE 410M-H) does not have two x16 slots. I have, however, one of those x1 to x16 riser cards you can get for quite cheap plugged into my board on one end and the ATI card on the other. I originally thought that the issue is either with that riser or something with the drivers not loading, but that seems not to be the case. To check that I ran

sudo lspci -vvv | grep -i -e VGA -e kernel

This produced the following (along with a bunch more lines):

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev ef) (prog-if 00 [VGA controller])
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV410 [Radeon X700] (rev ff) (prog-if ff)
	Kernel driver in use: radeon
	Kernel modules: radeon

This looks o.k. to me, as both the cards show up correctly and seem to have the appropriate driver in use according to this table on the wiki.

I've also tried custom config files in /etc/X11/xorg.conf.d/ with no success, I can post those if necessary.

Logs are of no use either, at least not that I can find. Quite literally nothing is written to any file in /var/log/ and

sudo systemctl status sddm.service

produces the following:

* sddm.service - Simple Desktop Display Manager
     Loaded: loaded (/usr/lib/systemd/system/sddm.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2021-10-30 14:43:43 CEST; 14min ago
       Docs: man:sddm(1)
             man:sddm.conf(5)
   Main PID: 378 (sddm)
      Tasks: 3 (limit: 19095)
     Memory: 17.2M
        CPU: 131ms
     CGroup: /system.slice/sddm.service
             |-378 /usr/bin/sddm
             `-624 /usr/lib/Xorg.wrap -nolisten tcp -background none -seat seat0 vt1 -auth /var/run/sddm/{00c5e2be-2ae9-4d1c-9aa7-b489b630d461} -noreset -displayfd 17

Oct 30 14:43:43 mashina systemd[1]: Started Simple Desktop Display Manager.
Oct 30 14:43:43 mashina sddm[378]: Initializing...
Oct 30 14:43:43 mashina sddm[378]: Starting...
Oct 30 14:43:43 mashina sddm[378]: Logind interface found
Oct 30 14:56:58 mashina sddm[378]: Adding new display on vt 1 ...
Oct 30 14:56:58 mashina sddm[378]: Loading theme configuration from ""
Oct 30 14:56:58 mashina sddm[378]: Display server starting...
Oct 30 14:56:58 mashina sddm[378]: Adding cookie to "/var/run/sddm/{00c5e2be-2ae9-4d1c-9aa7-b489b630d461}"
Oct 30 14:56:58 mashina sddm[378]: Running: /usr/bin/X -nolisten tcp -background none -seat seat0 vt1 -auth /var/run/sddm/{00c5e2be-2ae9-4d1c-9aa7-b489b630d461} -noreset -displayfd 17

I can see that sddm is not the issue, but X. This is further reinforced by the fact that if I do

xinit

myself, it first prints

waiting for X server to begin accepting connections .
..
..
..

and then doesn't do seemingly much of anything for a couple minutes (except for print a lot more ".." lines) before printing

xinit: giving up
xinit: unable to connect to X server: Connection refused

waiting for X server to shut down .........
xinit: X server slow to shut down, sending KILL signal

waiting for server to die ....
xinit: X server refuses to die

How can I get into an X11 session? Have I missed something crucial? Like I said, before adding a second card the system would boot into the login screen just fine and everything would work normally with either card (as long as the other one was not connected). Any and all help is appreciated.

P.S. This is my first post on the forum, please feel free to ask me to post anything you think is important that I haven't attached.
P.P.S. Although I am somewhat familiar with Linux I am completely new to Arch, so just bear with me please if I got something badly wrong.

Last edited by toljaga (2021-10-31 22:15:05)

seth · 2021-10-30 19:49:08

Which is just the last thing that shows up before I see the login screen anyway.

Remove the quiet parameter from the kernel commandline…

This looks o.k. to me, as both the cards show up correctly and seem to have the appropriate driver in use according

Just that the cards show up does not mean that they can be used, it might be underpowered or the GPU usage fails due to the limited throughput.
Have you tried using only one card and using that riser?

For further investiation, please post a complete system journal and ideally Xorg log for the failing boot.
"sudo journalctl -b -1" prints the journal for the previous boot (redirect it into a file or pastebin service, do not copy & paste out of the pager)
For the xorg log location see https://wiki.archlinux.org/title/Xorg#General - since it might easily get replaced when booting into X11 w/ only one GPU, you need to either preserve it in a different location or upload it:

cat /var/log/Xorg.0.log | curl -F 'f:1=<-' ix.io

toljaga · 2021-10-30 23:13:53

Hey, thank you for your reply.

Removing the quiet kernel parameter yields perhaps even less interesting results, namely no errors, just services starting successfully, here is the last couple lines

      Starting Permit User Sessions...
[ OK ] Started Permit User Sessions
[ OK ] Reached target Multi-User System.
[ OK ] Started Simple Desktop Display Manager.
[ OK ] Starting Permit User Sessions...
[ OK ] Reached target Graphical Interface
          Starting Hostname Service...
[ OK ] Started Hostname Service.

It shows sddm starting successfully.

It does boot fine with a single GPU and the riser installed. I'm not sure a card can fail due to low bandwidth, especially since these risers are designed with GPUs in mind (usually for mining), but I should note that the card I'd like to use in the riser (the X700) has a max power consumption of <45 W and runs solely off of PCI-e power; the riser itself has a 6-pin power connector where it gets the 75 W of power needed for PCI-e devices directly from the PSU, which is rated for 750 W meaning that it should power the system comfortably.

As you requested, here is the journalctl output.
However, no Xorg.n.log (or anything similar for that matter) is created when booting with both of the cards, either in /var/log/ or in /home/four/.local/share/xorg/ (<four> being my username), which are the locations listed as log folders in the link you posted. I know this for sure because I can still access tty2 (as well as tty3-6) when X hangs on tty1, which is how I gathered most of the info in the original post (I probably should have mentioned this earlier, sorry).

Thank you for your patience.

seth · 2021-10-31 07:35:40

However, no Xorg.n.log (or anything similar for that matter) is created

Boot the multi-user.target (2nd link below) and the run

Xorg 2>&1 | tee ~/xorg.not.log

and post that file.

Also, just a hunch

pacman -Qs xf86-video

toljaga · 2021-10-31 10:16:17

When booting the multi user target and running that it just hangs again. The xorg.not.log file remains empty. I've had it running for about half an hour now just to make sure and nothing happens.

Here's the output of the pacman command.

Doing some more digging, while the Xorg command was 'running' on tty1 I discovered that it seems to be stuck on Xorg.wrap. I ran this:

ps aux | grep tty1

which told me this

four         494  0.0  0.0   4628  3820 tty1     Ss   10:45   0:00 -bash
root        5136  0.0  0.0   2364   748 tty1     D+   10:46   0:00 /usr/lib/Xorg.wrap
four        5137  0.0  0.0   2392   688 tty1     S+   10:46   0:00 tee /home/four/xorg.not.log
four        5195  0.0  0.0   3256  1644 tty2     S+   11:13   0:00 grep --color=auto tty1

Just thought it might be of use. Tried searching around the internet for that specific hang and ended up downgrading the kernel but to no effect.

seth · 2021-10-31 12:38:49

Does anything get written into the /home/four/xorg.not.log ?
(Xorg might switch the TTY so you won't see it's immediate output, but we T-piped it there as well)

Edit: fuck (I forgot to look into the other tab I opened ;-)
So you've installed pretty much every xf86-video-* driver - remove them all (except maybe vmware)

I assume what happens is that the radeon driver tries to use the amdgpu card and induces a crash in the kernel module.
W/o the listed packages you'll be using the modesetting driver and that should play nicely w/ both chips.
If you want to pass through one GPU anyway, the local drivers won't "see" it and you can use them again - otherwise we'd likely have to bind the drivers to the GPUs.

Last edited by seth (2021-10-31 12:42:26)

toljaga · 2021-10-31 13:27:53

seth wrote:

Does anything get written into the /home/four/xorg.not.log ?
(Xorg might switch the TTY so you won't see it's immediate output, but we T-piped it there as well)

Nothing at all.

seth wrote:

So you've installed pretty much every xf86-video-* driver - remove them all (except maybe vmware)

Done, nothing seems to change after reboot.

seth wrote:

I assume what happens is that the radeon driver tries to use the amdgpu card and induces a crash in the kernel module.
W/o the listed packages you'll be using the modesetting driver and that should play nicely w/ both chips.

Modesetting does seem like a good option. From what I understand X should default to modesetting if no xf86-video-* drivers are present, but I don't know how to check if that's what is happening. lspcie -v still lists amdgpu and radeon (for the RX 570 and X700 respectively) as both the "kernel driver in use" and "kernel modules". Should I uninstall radeon (mesa) and/or amdgpu (amdvlk, mesa and mesa-vdpau) as well?

seth · 2021-10-31 14:30:08

Nothing at all.

That's bad…

From what I understand X should default to modesetting if no xf86-video-* drivers are present

Yes.

I don't know how to check if that's what is happening

It'd be in the Xorg log… but if you removed those packages, the modesetting driver *has* to be used anyway

lspcie -v still lists amdgpu and radeon (for the RX 570 and X700 respectively) as both the "kernel driver in use" and "kernel modules".

Yes, the kernel module is distinct from the X11 driver and this output is expected and ok.

[qutoe]Should I uninstall [mesa] and/or [amdvlk, mesa and mesa-vdpau] as well?[/quote}
No.

The easiest way to avoid this is likely to pass the GPU through (what's the end-goal here anyway, is it?)
https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF

You could also try to write a static server config - boot w/ only the radeon GPU in the riser slot and run "Xorg -configure" to generate one.
It should™ still work even when adding the amdgpu GPU, no idea whether that will allow Xorg to start.

In doubt we'll have to strace Xorg to see where it's hanging (but likely on the PCI enumeration)

toljaga · 2021-10-31 16:18:15

Okay I think this is a dead end.

I tried to boot with only the radeon GPU in the riser to try and config Xorg statically like you said and there was no output. I thought it was the riser again and tried only the amdgpu GPU in the riser but it worked perfectly fine and displayed everything normally (even booted into the desktop when I set the graphical.target again). O.K., so the radeon GPU must be dead, right?
Nope. It works fine if left alone in the x16 mobo slot. Boots into the desktop and everything.

Okay combination #3, with the radeon GPU in the x16 slot and the amdgpu GPU in the riser. This had not worked before, but for some reason now it does. (It's very funky and half of the right display is on the left one but at least some form of desktop is reached).

So my conclusion (if you can call it that) is that the X700 will not output anything while in the riser for the life of it. The riser and the card just seem to hate each other. Digging through lspci -vv some more I found the famous "!!! Unknown header type 7f"" error under the X700 while the card is in the riser yet nothing when in the mobo.

That means that even if I were to pass the other card through I'd still be stuck, right? I'm officially out of ideas.

seth · 2021-10-31 21:13:13

That means that even if I were to pass the other card through I'd still be stuck, right?

Unless you're willing to run the AMDGPU on the PCIe x1 …

Did you try the other x1 slot (not holding my breath, but hey…)
Does the CPU have an IGP (the board wires a D-SUB output)?

toljaga · 2021-10-31 22:10:24

Hey,
Running my main performance card in x1 is really not an option. Yes, I've tried the other x1 slot as well as just plugging the riser into the x16 slot but nothing changes. Also, no, I have no iGPU on the processor because I'm cheap like that. I'm completely shit out of luck.

Oh well, I guess it's back to boring shitty Windows for me. A shame really, because I actually really liked Arch for the last couple days that I did have it. I'll look into buying a used gt 710 or 1030 that may actually work with this project of mine, but we all know what the GPU market is currently like.

Either way, since this is no longer an Arch issue, I will be closing the thread, as soon as I figure out how to do that.

Thank you so much for your patience with me and your weekend time.

Edit: seems you can't really mark a thread closed here, so I'll mark it as solved.

Last edited by toljaga (2021-10-31 22:14:30)

Arch Linux

#1 2021-10-30 14:42:08

[SOLVED] X hangs when multiple GPUs are present

#2 2021-10-30 19:49:08

Re: [SOLVED] X hangs when multiple GPUs are present

#3 2021-10-30 23:13:53

Re: [SOLVED] X hangs when multiple GPUs are present

#4 2021-10-31 07:35:40

Re: [SOLVED] X hangs when multiple GPUs are present

#5 2021-10-31 10:16:17

Re: [SOLVED] X hangs when multiple GPUs are present

#6 2021-10-31 12:38:49

Re: [SOLVED] X hangs when multiple GPUs are present

#7 2021-10-31 13:27:53

Re: [SOLVED] X hangs when multiple GPUs are present

#8 2021-10-31 14:30:08

Re: [SOLVED] X hangs when multiple GPUs are present

#9 2021-10-31 16:18:15

Re: [SOLVED] X hangs when multiple GPUs are present

#10 2021-10-31 21:13:13

Re: [SOLVED] X hangs when multiple GPUs are present

#11 2021-10-31 22:10:24

Re: [SOLVED] X hangs when multiple GPUs are present

Board footer