Arch Linux

wikavalier

Member

Registered: 2013-01-22

Posts: 27

[SOLVED] from VFIO to NVIDIA driver, how to?

Hi all,

Is there any way to switch back to the nvidia drivers after setting them to be vfio WITHOUT rebooting?

uname -a
Linux demirusb 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux

My system is a gigabyte mobo with i7-4770 and nvidia GTX780ti.

I am using an script to switch pci drivers, part of it:

                modprobe $2
                # Ensure that the drivers will not run to take ownership of the bus
                # echo 0 > /sys/bus/pci/drivers_autoprobe

                # Gather relevant information of the device in BUS
                VENDOR=$(cat /sys/bus/pci/devices/0000\:$1/vendor)
                DEVICE=$(cat /sys/bus/pci/devices/0000\:$1/device)
                if [ -e /sys/bus/pci/devices/0000\:$1/driver ]; then
                        echo 0000:$1 > /sys/bus/pci/devices/0000\:$1/driver/unbind
                fi
                
                echo  $VENDOR $DEVICE > /sys/bus/pci/drivers/$2/new_id

                # line below should work as Alex Williamson indicates... but not for me
                #echo 0000:$1 > /sys/bus/pci/drivers/$2/bind
       
                echo   $VENDOR $DEVICE > /sys/bus/pci/drivers/$2/remove_id

                exit 0

The pcidriver script works fine with nouveau and vfio (also for snd_hda_intel and others) . Also the other way around

sudo lspci -kns 01:00.0
01:00.0 0300: 10de:100a (rev a1)
        Subsystem: 1458:3633
        Kernel driver in use: nouveau
        Kernel modules: nouveau, nvidia

pcidriver  01:00.0 vfio-pci
Changing 0000:01:00.0 device to vfio-pci driver

lspci -kns 01:00.0
01:00.0 0300: 10de:100a (rev a1)
        Subsystem: 1458:3633
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia

but if i try to use with nvidia. From vfio and nouveau

pcidriver  01:00.0 nvidia
Changing 0000:01:00.0 device to nvidia driver

lspci -kns 01:00.0
01:00.0 0300: 10de:100a (rev a1)
        Subsystem: 1458:3633
        Kernel modules: nouveau, nvidia

I tried blacklisting nouveau,rmmod nouveau, ttm and drm. DRM cant be unload as is used by the intelhd i915. Nothing works.

I checked dmesg and (because of the modprobe nvidia in the script) and

nvidia: probe of 0000:01:00.0 failed with error -1

The weird thing is that somehow the driver change is working. I can see that the udev is loaded (/dev/dri/card1) for the nvidia and i can also access the nvidia-smi.

Questions:

1. Why is this line not working?:

echo 0000:$1 > /sys/bus/pci/drivers/$2/bind

I am following this mail from AW:

https://lists.linuxfoundation.org/piper … 24718.html

2. Shall I assume that once nvidia-smi and /dev/dri/card1 are in place, the driver is loaded? Why then it is not shown in the lspci -kns 01:00.0?

3. I am using latest nvidia drivers. I also tried nvidia-kms and rebuilding initramfs with the same results.

4.  Where are the controlDxx and renderDxx for the nvidia? I rode that those udev files are not created because of the privative driver. Am I right? Therefore, if i made an lxc granting access to card1 (nvidia udev) will i have everything in place to render in the lxc?

Cheers!

W

EDIT--------------------------------------
I found out that a certain combination of rmmod and modprobe will make it work.

If I am using the vfio-pci i need to:
rmmod vfio-pci
modprobe nvidia

I am still debugging what is going on here as, if i do something different it wont work again till reboot.

EDIT2---the solution------------------------------

I finally I found out the best and most reliable way to move from nvidia to vfio and vice versa.... going through nouveau!!

so if you are in nvidia:
1. rmmod nvidia
2. modprobe nouveau
3. use the script to use vfio-pci

If you are in vfio-pci:
1. Use the script to load nouveau
2. rmmod nouveau
3. modprobe nvidia

It worked 100% of the times i tested.

Last edited by wikavalier (2015-11-06 07:34:48)