[SOLVED] Nvidia dual gpu stack config question

Succulent of your garden · 2025-05-22 16:26:41

Hi!

I'm considering buying a cheap nvidia RTX 5000 series [yes, I know they are not so much better than 4000 series, but I want the 5th generation tensor cores, it's not for gaming it's for work] because I'm starting to do many machine learning stuff with deep learning libraries. But I wan't to have my PC with a dual stack gpu setup, with a pascal based gpu.

As you can see in the https://wiki.archlinux.org/title/NVIDIA the packages for the drivers are different for each gpu.

So first question: Is Arch currently supporting the RTX 5000 series with the nvidia-open and nvidia-open-lts packages ?
and the second question: Can i have both gpus running in my system right ? With both packages I'm assuming that the kernel will know what to do with each gpu. I want to have the second one to do gpu pass-through to virtual machines.

Last edited by Succulent of your garden (2025-05-24 23:40:06)

Xephon · 2025-05-22 17:00:25

RTX 5000 cards are supported by nvidia-open and nvidia-open-lts.

No you can't have both Pascal and Blackwell running simultaneously. The former needs nvidia package, the latter - nvidia-open and those are in direct conflict with each other. Meaning that during the installation of one of them pacman will automatically remove the other one.

Succulent of your garden · 2025-05-22 17:23:48

But in the worst scenario can I just use the nouveau driver for the pascal based with the blackwell using the nvidia driver ? I mean, the pascal one doesn't have tensor cores either, so I will use it for VM only use.

Xephon · 2025-05-22 17:38:14

https://wiki.archlinux.org/title/NVIDIA

The nvidia-utils package contains a file which blacklists the nouveau module once you reboot

You can't have two different nvidia drivers running simultaneously on the same system.

Succulent of your garden · 2025-05-22 20:37:28

thanks for the info. So there is no any chance to un-blacklist the nouveau driver so I can get my pascal gpu working together ?

Why is this the case ? Does the same happen with amd cards ?

Xephon · 2025-05-22 21:28:59

Nouveau is blacklisted for a reason. If you un-blacklist it, nouveau will load instead of nvidia-open.

There is no chance to run two different drivers. Period. Your assumption that the kernel will know what to do with each gpu is not based on reality.

loqs · 2025-05-22 21:56:16

You could pass one of the cards through to a virtual machine to use avoiding having conflicting kernel modules loaded by the same kernel.

seth · 2025-05-23 06:58:48

Also

The OP wrote:

I want to have the second one to do gpu pass-through to virtual machines.

What makes the entire thread moot.

Succulent of your garden · 2025-05-23 16:05:24

Okey. Let summarize everything:

1) Currently I'm having amd + nvidia stack in the same PC. I made programs that run in the same OS and environment using each one of the gpus, and both at the same time, with or without the use of ROCm and CUDA in parallel. So for that reason I believe that in practice the kernel can handle more than one driver for a gpu. So this is an issue only with nividia or when you are running a stack with all the cards from the same manufacture but one card is much older than the new one ?. So the first question is: If I have a old Radeon card and make a setup with a newer one (The complete opposite of what I'm trying to do) this will work ? This question is made in mind using the context without the virtual machines, just os being capable of using both gpus at the same time, in this case a very old Radeon with a newer one. Does the same happen ?

2) So it is possible that I can use my pascal nvidia gpu in the virtual machine ? I mean I can create a VFIO passthrough and put the driver in the vm right ? I'm seeing this setup like this:

2.1) Host OS: Okey, when the user uses lspci I can see there is second nvidia card, but I can't use it, since I don't have a driver for that.
2.2) VM: Oh I have a gpu and the driver, since the user is giving me access to the pci directly since I have the IOMMU nice.

Is this scenario possible with the Blackwell architecture as primary gpu right ? If that's the case: Does the Blackwell gpu could have issues in rendering the video output of the pascal base gpu in my virtual machine ? Since the hdmi or displayport cable of the screen will be connected to the blackwell only. If I'm not remembering wrong, the amd gpus are more friendly to the VFIO passtrhough when they are the host gpu, and the nvidia are great for sending the gpu to the vm. But does anyone have tried a setup like what I maybe going to do ?

Last edited by Succulent of your garden (2025-05-23 16:08:25)

Xephon · 2025-05-23 18:44:10

1)

So this is an issue only with nividia or when you are running a stack with all the cards from the same manufacture but one card is much older than the new one ?

It's an issue when you are running a stack containing some cards from the same manufacturer that require different drivers and those drivers are in conflict with each other. It doesn't happen with AMD cards: amdgpu and radeon kernel modules are not in conflict and can be used together. Moreover amdgpu supports pretty old cards, so you might not even need radeon driver.

2) You definitely can use Pascal in VM (two different drivers in two separate systems). But using the same monitor for both host and VM could be tricky. Might need something like https://looking-glass.io/

seth · 2025-05-23 19:21:37

Does the monitor have multiple inputs and the ability to switch between them?

The moment you add the gpu to vfio the host OS does no longer see it and whatever renders the output inside the VM, the host doesn't care. It renders whatever the VM gives it.
Don't overcomplicate this, vfio forwarding is covered extensively in the wiki.

Succulent of your garden · 2025-05-24 15:00:40

many thanks both of you in your answers. I'm going to use looking glass, but as I know the nvidia host gpu is not recomended by the following reasons:

AMD or Intel for the client
AMD and Intel both support the DMABUF feature which enables offloading memory transfers to the GPU hardware. Please note that making use of this feature requires loading the KVMFR kernel module.
Additionally AMD GPUs suffer stability issues when operating as a passthrough device and as such we do not recommend their usage for such purposes. Models of note that have issues include but are not limited to the entire Polaris, Vega, Navi and BigNavi GPU series. Vega and Navi are notably the worst and should be avoided for virtualization usage.
NVIDIA for the guest
NVIDIA unlike AMD do not seem to suffer from the same stability issues as AMD GPUs when operating as a passthrough GPU, however due to the closed source nature of their drivers NVIDIA can not make use of the DMABUF feature in the Linux kernel unless you use the open source NVIDIA drivers.

source: https://looking-glass.io/docs/B7/requirements/

But as far as I know the Blackwell architecture is currently having the drivers open source right ? Not sure if the DMABUF is needed also for the pascal gpu, since the drivers are closed source. As I'm seeing it right now, if I get the blackwell based gpu I could use looking glass without any problems.

What do you think about this ? Does any one have tried something like this ? maybe not with blackwell gpu but with two nvidia gpu stack.

EDIT: My monitor does have multiple inputs and is possible to switch between them.

Last edited by Succulent of your garden (2025-05-24 15:27:56)

Xephon · 2025-05-24 17:15:05

No one can guarantee that you won't have any problems, but it looks like you're gonna be fine.

Looking Glass developers confirm that nvidia-open allows to use foreign DMABUF objects. Should be no problems using Blackwell for the host.
https://github.com/NVIDIA/open-gpu-kern … t-11022142

And Pascal in VM won't need DMABUF support capabilities

Succulent of your garden · 2025-05-24 23:39:43

Thanks very much for your help Xephon, and also for the info by Seth. Really appreciated

Arch Linux

#1 2025-05-22 16:26:41

[SOLVED] Nvidia dual gpu stack config question

#2 2025-05-22 17:00:25

Re: [SOLVED] Nvidia dual gpu stack config question

#3 2025-05-22 17:23:48

Re: [SOLVED] Nvidia dual gpu stack config question

#4 2025-05-22 17:38:14

Re: [SOLVED] Nvidia dual gpu stack config question

#5 2025-05-22 20:37:28

Re: [SOLVED] Nvidia dual gpu stack config question

#6 2025-05-22 21:28:59

Re: [SOLVED] Nvidia dual gpu stack config question

#7 2025-05-22 21:56:16

Re: [SOLVED] Nvidia dual gpu stack config question

#8 2025-05-23 06:58:48

Re: [SOLVED] Nvidia dual gpu stack config question

#9 2025-05-23 16:05:24

Re: [SOLVED] Nvidia dual gpu stack config question

#10 2025-05-23 18:44:10

Re: [SOLVED] Nvidia dual gpu stack config question

#11 2025-05-23 19:21:37

Re: [SOLVED] Nvidia dual gpu stack config question

#12 2025-05-24 15:00:40

Re: [SOLVED] Nvidia dual gpu stack config question

#13 2025-05-24 17:15:05

Re: [SOLVED] Nvidia dual gpu stack config question

#14 2025-05-24 23:39:43

Re: [SOLVED] Nvidia dual gpu stack config question

Board footer