Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen it in several posts recently, so I'll mention again, if you're running qemu.git you should not need to build and run a separate seabios.  Only long, long ago before qemu updated their seabios snapshot was this necessary.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

trimm

Member

Registered: 2014-03-06

Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another thing though, and its my sound. I gave up on passing through my USB headset(crackling noise) and sharing sound between my host and guest. (I tried following the first post without success)
Im thinking I can passthrough my integrated sound on the motherboard, and I tried doing it like this:

Binding the sound:

vfio-pci 0000:00:1b.0

In my dmesg I get the following:

vfio-pci 0000:00:1b.0: kvm assign device

And in my script I have this:

-device vfio-pci,host=00:1b.0

No errors in Linux, but I get a red mark over my audio icon in Windows. Going into "sound" tab inside audio properties and the whole VM totally freezes.

I passthroughed with vfio-pci, a drive and my secondary network card without any issues, however I have not had any success with audio.
I have also passthroughed my Xbox Controller and Rocksmith USB adapter.

Any help is appreciated!

I haven't had any trouble passing HDA audio from the chipset through to the guest.  I notice a discrepancy in your process though, the string "kvm assign device" is only printed out by virt/kvm/iommu.c:kvm_assign_device() which is only called if using pci-assign instead of vfio-pci.  AFAIK either will work, but I don't know what to trust in your report...

You are quite observant aw, I posted my journalctl when I tried pci-assign.
This is the message I get when I use vfio-pci:

vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002)

I just tested my VM via Virtual Manager GUI, passed through my audio(0000:00:1b.0) and I got sound working.
However the following command is not giving me any audio in Windows. What am I doing wrong? (The last line is my audio..)

sudo -E qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 16384 \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw,cache=none \
-device ide-hd,bus=ahci.0,drive=disk \
-device vfio-pci,host=00:19.0,bus=pcie.0 \
-drive file=/dev/sdb1,id=mmo,format=raw,cache=none \
-device ide-hd,bus=ahci.1,drive=mmo \
-drive file=/home/thor/Windows/Windows-Steam.img,id=steam,format=raw,cache=none \
-device ide-hd,bus=ahci.2,drive=steam \
-device vfio-pci,host=00:1b.0

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sudo -E qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 16384 \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw,cache=none \
-device ide-hd,bus=ahci.0,drive=disk \
-device vfio-pci,host=00:19.0,bus=pcie.0 \
-drive file=/dev/sdb1,id=mmo,format=raw,cache=none \
-device ide-hd,bus=ahci.1,drive=mmo \
-drive file=/home/thor/Windows/Windows-Steam.img,id=steam,format=raw,cache=none \
-device ide-hd,bus=ahci.2,drive=steam \
-device vfio-pci,host=00:1b.0

Why are you passing through both the GPU audio device 1:00.1 and the motherboard device 00:1b.0 if you only plan to use the latter?  Simplify the problem and drop GPU audio.  It's possible Windows is confused about which device to use.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

trimm

Member

Registered: 2014-03-06

Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw: I just found out the weirdest thing.
If I put the audio pass through down at the bottom, it wont work.
If I put it right beneath -nographic, it works!

I cant see the logic, perhaps because of the disks are passed through and it takes too much time or something.

May be good to know if any of you experience issues with something not working and is in a situation similar to mine.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.

Uhm ... now I am *very* glad, that flashing the bios of my gtx 660 worked

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.

Uhm ... now I am *very* glad, that flashing the bios of my gtx 660 worked

You people are nuts

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Jesse2004

Member

Registered: 2012-06-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all.

I have a newbie's question: why using two scsi-cds in virtio-scsi-pci does not work?

That is, when using this commandline,

qemu-system-x86_64  ......  -device virtio-scsi-pci,id=scsi0  \
        -drive file=win8.iso,id=cd1  -device scsi-cd,drive=cd1  \
                -drive file=virtio_driver.iso,id=cd2  -device scsi-cd,drive=cd2

during windows installation, the drive containing "virtio_driver.iso" will not show up (while on the boot screen, press f12, there are two cdroms there). Why is that? (I'm using qemu 1.7.0 in the official repository and I know using a ide-cd will work, just want to know more about how virtio-scsi-pci works).

Is there any limitation on the number of device on virto-scsi-pci? These commandlines of virto-scsi-pci are just mysterious to me, can't find anything about it in the manpage and  `qemu-system-x86_64 -device virtio-scsi-pci,help`  just shows some succinct words. Can anyone please point me to some documentation?

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get some older Radeon (like HD 3870) to work with Windows? Is there some known reason why only 5000+ Radeons are reported as working? Am I trying to do something hopeless? (If so I'd immediately go buy a new card.)

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

[ 9367.516194] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
[ 9389.765260] ata2: illegal qc_active transition (0000000f->93e80875)
[ 9389.774688] ata2.00: exception Emask 0x2 SAct 0xf SErr 0x0 action 0x6 frozen
[ 9389.774695] ata2.00: failed command: READ FPDMA QUEUED
[ 9389.774704] ata2.00: cmd 60/10:00:80:b3:08/00:00:fa:00:00/40 tag 0 ncq 8192 in
         res 40/00:18:a8:b4:08/00:00:fa:00:00/40 Emask 0x2 (HSM violation)
[ 9389.774707] ata2.00: status: { DRDY }

I have seen identical messages with Win 7 too. This time I managed to kill qemu myself. It seems those errors might go away if launching from console without graphics, but it won't improve anything...

Last edited by ahl (2014-03-18 06:23:17)

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

heads-up: 3.14 rc7 is out. There is some stuff we might find interesting:
https://www.kernel.org/diff/diffview.cg … .xz;z=2534
https://www.kernel.org/diff/diffview.cg … .xz;z=2539
https://www.kernel.org/diff/diffview.cg … .xz;z=2540
https://www.kernel.org/diff/diffview.cg … .xz;z=2541

Will be testing later today.

fukhda

Member

Registered: 2014-02-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Good news here, a success VGA passthrough in Fedora 20. In my very first post , I swamped in code 12 issue. Finally it is solved. Le tme explain the process in the below.

1. When the assigned AMD card and QXL vga card coexists in VM (Win7x32), there would be a conflict causing AMD card a code12. (AMD driver to be installed here)

2. By removing the QXL vga (add the swithc -vga none in the script launching VM), there would be a black screen in QEMU window.

3. Attach another monitor to the AMD card, the VM Win7 is succesfully displayed.

4. Pass through (via vfio) a Renesas USB3.0 Controller with Keyboard/mouse connected, otherwise we can not operate VM. (Renesas driver to be installed here)

5. Check the device manager for AMD card, no more code 12. Now, enjoy the VGA passthrough.

Passthrough NVIDIA EGVA GTX650 works too. Thanks for everyone, and especially for nbhs.

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Good news here, a success VGA passthrough in Fedora 20. In my very first post , I swamped in code 12 issue. Finally it is solved. Le tme explain the process in the below.

1. When the assigned AMD card and QXL vga card coexists in VM (Win7x32), there would be a conflict causing AMD card a code12. (AMD driver to be installed here)

2. By removing the QXL vga (add the swithc -vga none in the script launching VM), there would be a black screen in QEMU window.

3. Attach another monitor to the AMD card, the VM Win7 is succesfully displayed.

4. Pass through (via vfio) a Renesas USB3.0 Controller with Keyboard/mouse connected, otherwise we can not operate VM. (Renesas driver to be installed here)

5. Check the device manager for AMD card, no more code 12. Now, enjoy the VGA passthrough.

Passthrough NVIDIA EGVA GTX650 works too. Thanks for everyone, and especially for nbhs.

oh you tried emulated vga adapter side by side with dedicated gpu.. interesting. i tried that too and with stock qemu from repositories i also had some error code in task manager. not sure if it was 12. after i compiled qemu from git repo that code was gone (this is still with -vga std -sdl). then qemu window displays everything just fine and dedicated gpu appears to be fine. however i was not able to run anything with full 3d acceleration. not that i tried much anyway. idk how useful this info could be to you... but yeah for full 3d acceleration i have to use -vga none and control VM with keyboard/mouse plugged into passed-through usb port just like you said.

rawuza

Member

Registered: 2013-12-22

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get some older Radeon (like HD 3870) to work with Windows? Is there some known reason why only 5000+ Radeons are reported as working? Am I trying to do something hopeless? (If so I'd immediately go buy a new card.)

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

Norcoen

Member

From: Norway

Registered: 2013-11-02

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

That might be the case, except the script in the first post does unbind first (if I am reading it correctly) and it was hanging on my video card when it ran (without the driver being blacklisted).  Whether or not pci-stub was grabbing the USB devices I don't know, but the posted script was unbinding them as expected (Confirmed when I redid my boot line and forgot to disable the service only to boot to a system without a working mouse/keyboard).

Regardless, I may have not done the best kernel config when I built mine.  Blacklisting definitely works and pci-stub is still magic to me.  And none of this conversation is probably of much use to AKSN74

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant for the life of me sort out my audio for my VM.

I use this:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

I have also tried with -soundhw all and -soundhw ac97.

This is what I get when I fire up the computer:

pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `adc'

In the first post its stated that I may have to use the QEMU_AUDIO_DRV variable. I have tried this (export QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_DRV=pa as root) without success. Anyone in the same boat as me, and/or found a solution?(except passing a USB headset? )

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted...

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

Hmm... that's interesting, I'm still a Linux newbie so these settings to me may can't understand. But

I saw the usage about softdep, maybe this is a best choice to replace the grub function. Only a problem is can softdep support pci-stub command (like echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind)

Because if I use grub, it will according from HID, that means both of GTX480 will bind into pci-stub and host will not have any GPU to display.

So my idea is, if I can bind 2nd GTX480 at first, then Nouveru will not attach 2nd GTX480. After that everything will be easy.

Is that possible?

Last edited by AKSN74 (2014-03-18 15:09:05)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

Hmm... that's interesting, I'm still a Linux newbie so these settings to me may can't understand. But

I saw the usage about softdep, maybe this is a best choice to replace the grub function. Only a problem is can softdep support pci-stub command (like echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind)

Because if I use grub, it will according from HID, that means both of GTX480 will bind into pci-stub and host will not have any GPU to display.

So my idea is, if I can bind 2nd GTX480 at first, then Nouveru will not attach 2nd GTX480. After that everything will be easy.

Is that possible?

The point of softdep is that it just orders the drivers, so you can have both pci-stub and your regular GPU drivers. You order it such as to load GPU driver second because they are "greedy" (like all other drivers) and will grab all matching devices, thus preventing pci-stub from doing its work. So, you understand it right : set softdep to ensure pci-stub "grabs" 2nd GFX480 first, and if Nouveau is loaded after this, it will not use this card. It might be easier to build kernel with CONFIG_PCI_STUB=y (but without Nouveau) in which case pci_stub will be guaranteed to be loaded first without use of softdep.

See also post #93, using modprobe/*conf rather than kernel command line parameter . I am not sure which one is better.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The point of softdep is that it just orders the drivers, so you can have both pci-stub and your regular GPU drivers. You order it such as to load GPU driver second because they are "greedy" (like all other drivers) and will grab all matching devices, thus preventing pci-stub from doing its work. So, you understand it right : set softdep to ensure pci-stub "grabs" 2nd GFX480 first, and if Nouveau is loaded after this, it will not use this card. It might be easier to build kernel with CONFIG_PCI_STUB=y (but without Nouveau) in which case pci_stub will be guaranteed to be loaded first without use of softdep.

See also post #93, using modprobe/*conf rather than kernel command line parameter . I am not sure which one is better.

Thanks for your hint, now I know how to do.

I thought that it still a big challenge to do my build, because HID is a big problem, I need to do pci-stub command after pci-stub load up and Nouveau not load up yet.

I'm thought about rc.local to do this (because GRUB can't do bash code) , but I found some information, rc.local launched after driver loaded so it's not an option.

Although I can't use both GTX480 for host and guest, I still can pass-through them into VM with softdep.

I'll try for this, and hope it work, thanks for your help.

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested 3.14-rc7 kernel. Everything works good. There is even performance increase for that passmark directx9 complex test. rc6 - 16 FPS, rc7 - 21 FPS.

P.S. In case noone noticed dr6 patch is already upstream this no longer needed. Adding my compilation of patches should anyone need. Usage - extract them to parent dir of kernel source code and from kernel source code dir run ../patch_kernel.sh

http://paste2box.com/5/#/3422361273/PsZ … hes.tar.gz

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

So I replaced my HD 3870 with a HD 6670. Nothing changed. Random errors continued. I wonder if HD 6670 is broken too?

Next I tried physically using different slots for all cards. Nothing changed.

And then I finally made some progress:
Removal of the cheap PCI-E network interface cards that I had on host resulted in tremendous improvement on system stability! Win 7 started working seemingly perfectly. I got graphics working for the first time and was able to run the system benchmark with good results.

For a while I thought it fixed everything, but on about 10th identical(snapshot=on) reboot of Windows 7 guest, I got some minor ATA errors on host again. However, all of the other randomness is gone. I wonder if the SATA controller is bad too (AMD A85X FCH)?

Next I tried WinXP. It seems to be much harder to boot than Win 7. It took a while and a whole lot of minor ATA errors until it finished. But it booted! So for the first time I also got WinXP running. Then I noticed a connection between qemu and those ATA errors: If I run qemu with snapshot=on I only get READ errors. Without snapshot I also get WRITE errors. So I moved WinXP completely in ramdisk and booted it: The exact same snapshot of WinXP that took a long time to boot from disk, now booted in just a few seconds. All of the ATA errors were gone too! Makes sense.

Any hints on what to try next? (I set my SATA controller to IDE mode now for testing. It used to be AHCI.)

Edit: By the way I wouldn't be surprised if a lot of people had this problem I have now. It could easily go unnoticed with Win 7. Also my HD is pretty slow, which could increase the chance of errors.

### Group 0 ###
    00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Complex
### Group 1 ###
    00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Trinity [Radeon HD 7660D]
    00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Trinity HDMI Audio Controller
### Group 2 ###
    00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series]
### Group 3 ###
    00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
    00:10.1 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
### Group 4 ###
    00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 40)
### Group 5 ###
    00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
    00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
### Group 6 ###
    00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
    00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
### Group 7 ###
    00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 14)
    00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD] FCH IDE Controller (rev 40)
    00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
    00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40)
### Group 8 ###
    00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0)
    00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 1)
    00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2)
    04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
    05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09)
### Group 9 ###
    00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 0
    00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 1
    00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 2
    00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 3
    00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 4
    00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 5

Last edited by ahl (2014-03-19 21:17:03)

noctavian

Member

Registered: 2013-07-11

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows-7-Main
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Windows-7-Main</name>
  <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='6' threads='6'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/windows_7_main.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fe:3e:b0'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>