Arch Linux

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now it gives me a different error

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

Last edited by Chetyre (2013-10-31 00:46:00)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now it gives me a different error

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

sabzn

Member

Registered: 2013-10-31

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys.
Sorry to bother you, but I just can't get this to work as it should.
For quite some time I've been trying to get Windows 7 to work with VGA-Passthrough using my 7970.
The VM starts fine and VGA-Passthrough seems to be working to some extend (the VM's output is shown through the passthrough'd GPU). Dmesg shows no errors either.
As I said, the video output through the VM works fine as long as I don't install the Catalyst drivers.
Whenever I try to install them, the VM BSOD's during the next restart with the "device reset timeout" error mentioned on page 23/24 in this thread, but the suggested fixes didn't work for me.
When I manually load the VGA BIOS (with rombar=0,romfile=/home/user/7970.rom), the VM does not bluescreen, but the Catalyst driver throws an error message saying that it failed to load the driver.
The GPU itself seems to be recognized by the Device Manager and programs like GPU-z, but there is still no acceleration available (see link below)
http://picpaste.com/pics/qlq1Pz79.1383184744.png

So far I have tried Catalyst 12.4, 12.10, 13.4 and 13.11 beta, but all of them show the same results.

Specs:
Intel i5-4670, integrated GPU used for the host
AsRock z87 Pro4
AMD Radeon HD 7970, used for passthrough
Kernel 3.12-rc6 with the i915 patches applied.

I have yet to apply the pcie_acs_override patch, but I don't think the problem is related to this one.

If you need any other information about the system such as exact configurations or logs, please let me know.

best regards

Last edited by sabzn (2013-10-31 02:52:00)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys.
Sorry to bother you, but I just can't get this to work as it should.
For quite some time I've been trying to get Windows 7 to work with VGA-Passthrough using my 7970.
The VM starts fine and VGA-Passthrough seems to be working to some extend (the VM's output is shown through the passthrough'd GPU). Dmesg shows no errors either.
As I said, the video output through the VM works fine as long as I don't install the Catalyst drivers.
Whenever I try to install them, the VM BSOD's during the next restart with the "device reset timeout" error mentioned on page 23/24 in this thread, but the suggested fixes didn't work for me.
When I manually load the VGA BIOS (with rombar=0,romfile=/home/user/7970.rom), the VM does not bluescreen, but the Catalyst driver throws an error message saying that it failed to load the driver.
The GPU itself seems to be recognized by the Device Manager and programs like GPU-z, but there is still no acceleration available (see link below)
http://picpaste.com/pics/qlq1Pz79.1383184744.png

So far I have tried Catalyst 12.4, 12.10, 13.4 and 13.11 beta, but all of them show the same results.

Specs:
Intel i5-4670, integrated GPU used for the host
AsRock z87 Pro4
AMD Radeon HD 7970, used for passthrough
Kernel 3.12-rc6 with the i915 patches applied.

I have yet to apply the pcie_acs_override patch, but I don't think the problem is related to this one.

If you need any other information about the system such as exact configurations or logs, please let me know.

best regards

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sabzn

Member

Registered: 2013-10-31

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,
Could you please help to find the best solution to share drive (array) between two virtual machines in the same time:
1. Real machine: qemu host, combine real hard drive to ZFS partition
2. Virtual Linux machine should has access to ZFS partition
3. Virtual Windows machine should has access to ZFS partition
I am sorry for offtop question, but it would be great if someone point me to right direction.

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

I had the same "Operation not permitted" issue, and that fixed it. Thanks!

Having it in libvirt is pretty neat! (Until now I had a systemd --user service)

Last edited by teekay (2013-10-31 12:32:53)

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,
Could you please help to find the best solution to share drive (array) between two virtual machines in the same time:
1. Real machine: qemu host, combine real hard drive to ZFS partition
2. Virtual Linux machine should has access to ZFS partition
3. Virtual Windows machine should has access to ZFS partition
I am sorry for offtop question, but it would be great if someone point me to right direction.

zfs set sharesmb=on pool/fs maybe?

http://zfsonlinux.org/faq.html#HowDoISetupShares

Last edited by teekay (2013-10-31 12:40:04)

sabzn

Member

Registered: 2013-10-31

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.

I just checked again and it is indeed using q35. The CPU-z Mainboard tab shows the same output as in OP's screenshots.

Do you think that this error might be related to the lack of the pcie_acs_override patch?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.

I just checked again and it is indeed using q35. The CPU-z Mainboard tab shows the same output as in OP's screenshots.

Do you think that this error might be related to the lack of the pcie_acs_override patch?

No.  Even if you're using q35 you're still placing the GPU on the root complex rather than behind a root port.  It would be on something other than bus 0 if it were behind a root port.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

I had the same "Operation not permitted" issue, and that fixed it. Thanks!

Having it in libvirt is pretty neat! (Until now I had a systemd --user service)

Guess I spoke too soon.

I can either get VFIO working, OR the ICH9 audio emulation with pulseaudio. But not both.

If I configure qemu to run as my main user, the VM starts and I get output on the passed through card, but the resolution is low, and also the passed through USB3 controller isn't working (so, no mouse -.-). The /dev/vfio/* devices are chown'ed.
In Qemu log I see

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f0fe95d3ce0, 0x0, 0xb0000000, 0x7f0e44000000) = -12 (C
annot allocate memory)
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f0fe95d3ce0, 0x100000000, 0xd0000000, 0x7f0ef4000000) 
= -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0x0, 0xc0000, 0x7f0e44000000) = -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0xc0000, 0x20000, 0x7f0fcf600000) = -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0xe0000, 0x20000, 0x7f0fdc200000) = -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0x100000, 0xaff00000, 0x7f0e44100000) = -12 (Cannot allocate memory)
...

And dmesg many of those

[ 9507.459310] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

and some

[ 8845.605035] AMD-Vi: Event logged [IO_PAGE_FAULT device=05:00.0 domain=0x001f address=0x00000001c8cec800 flags=0x0000]
[ 8847.018681] AMD-Vi: Event logged [IO_PAGE_FAULT device=04:00.0 domain=0x001f address=0x00000001bf330000 flags=0x0000]

If I run as root, VFIO works fine, but it cannot connect to my user's pulseaudio session:

pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver

My domain .xml has

    <qemu:env name='QEMU_PA_SAMPLES' value='128'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>

I must be missing something, either way.

Last edited by teekay (2013-10-31 19:25:07)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Teekay,

Pulseaudio is running as a your user, qemu running as root may have difficulties to connect to its session i guess.

I stopped bothering with pulseaudio and qemu's sound emulation, as i couln't get the a/v latency below an acceptable minimum.
When playing e.g. this A/V Sync test http://www.youtube.com/watch?v=cGgf_dbDMsw - i always had a noteable lag (with above settings QEMU_PA_SAMPLES=128 it was minimum lag - but still noteable).
So i gave up with a fully virtualized soundcard and use a USB connected one.

Regarding mouse support i can recommend synergy: Its fast enough even for gaming, and when you press the Scroll-Lock button, the mouse gets locked which helps many fullscreen-applications to function perfectly with synergy.

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Teekay,

Pulseaudio is running as a your user, qemu running as root may have difficulties to connect to its session i guess.

I stopped bothering with pulseaudio and qemu's sound emulation, as i couln't get the a/v latency below an acceptable minimum.
When playing e.g. this A/V Sync test http://www.youtube.com/watch?v=cGgf_dbDMsw - i always had a noteable lag (with above settings QEMU_PA_SAMPLES=128 it was minimum lag - but still noteable).
So i gave up with a fully virtualized soundcard and use a USB connected one.

Regarding mouse support i can recommend synergy: Its fast enough even for gaming, and when you press the Scroll-Lock button, the mouse gets locked which helps many fullscreen-applications to function perfectly with synergy.

Hi kaeptnb,

thanks for your input.

When using qemu command line (as user), both vfio and ICH9 emulation to pulseaudio works fine. For mouse I pass through a dedicated USB3 controller and use a KVM-switch, so no issues there. Audio is working fine for what I need it, too.

I just wonder what's missing to get VFIO fully working when running as user in libvirt. I did
* chown tom:tom /dev/vfio/{17,18,19}
* chmod 666 /dev/vfio/vfio
* set memlock to 6GB + 2GB overhead in limits.conf (the VM is confiured to run with 6GB)
* set user, group and "clear_emulator_capabilities = 0" in /etc/libvirt/qemu.conf

Still same RLIMIT_MEMLOCK and "vfio_dma_map .. Cannot allocate memory" errors.

I'd really like to run as user, not root - Audio is not the only issue here (I could just enable the onboard HDA, pass it through and plug it into my host MAUDIO Delta's input) - but,  running Windows as root sounds scary!

Last edited by teekay (2013-10-31 21:48:12)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes you're right, the need to run it as root is suboptimal, but to be honest i think IF any windows process is capabale to get out of kvm its already on the kernel-level - so getting root may be easy anyhow.

But yes - i also would like to run libvirt/qemu as non-root...

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh i overread the main part: you can sucessfully run qemu as a user manually!
that never worked for me, when i startet qemu as a user i got the same or similar messages like you.

Last edited by kaeptnb (2013-10-31 22:52:38)

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh i overread the main part: you can sucessfully run qemu as a user manually!
that never worked for me, when i startet qemu as a user i got the same or similar messages like you.

Yup, that works fine. It just required to chown the /dev/vfio devices and the hugetlbfs area.

Having it in libvirt would be cosmetic. I have some LXC containers managed with libvirt, so this would put everything in a unique interface. But libvirt causes me a headache, no idea where it takes the RLIMIT_MEMLOCK value from. It seems to ignore limits.conf

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

but the shell magic to automise this is probably more hassle than to adapt to libvirt

Here is a script I use to pin vcpus. It works, though it's ugly:

qemu [your usual options] -qmp unix:./qmp-sock,server,nowait --monitor stdio

i=0
echo '{"execute":"qmp_capabilities"} {"execute":"query-cpus"}' | socat UNIX-CONNECT:./qmp-sock STDIO | grep -Po '"thread_id": (\d)*' | awk '{print $2}' | while read tid
do
       taskset -pc $((i++)) $tid
done

Now a question regarding libvirt.
Is it possible to launch just one vm with root privileges in libvirt? All my other vms work fine with regular user privileges, I don't want to switch all of them to root because of one special vm.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

libvirt does have vfio support, so many of you may have any easier time getting the latest libvirt and enabling vfio that way.  Then it's a matter of a small wrapper script to add the ,x-vga=on option for the VGA device.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got a bit further regarding the libvirt-as-user issue. On the libvirt mailing list there were some comments about recently added patches the make libvirt set a proper RLIMIT_MEMLOCK value, if either memory is locked in the domain, or if VFIO is configured somewhere.

This is were your comment comes in, @aw

libvirt does have vfio support, so many of you may have any easier time getting the latest libvirt and enabling vfio that way.  Then it's a matter of a small wrapper script to add the ,x-vga=on option for the VGA device.

I'm using latest libvirt-git, and I'm struggling to convert this

-device vfio-pci,host=03:00.0,bus=pcie.0

into libvirt xml.
Using "virsh edit" I tried like this:

    <hostdev type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

which generated this address
<address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
and this command line:

-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.2,addr=0x6

It made the RLIMIT_MEMLOCK issues go away, but it also made the VM run into a boot->crash->boot loop.

Not sure if I'm doing something wrong.. is there a pcie address type maybe?

Norcoen

Member

From: Norway

Registered: 2013-11-02

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I Have a problem passing through my PCIe Combo Card to the virtual machine, because it has the same ASM1062 Serial ATA Controller as I have onboard.

# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5)
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 [8086:8c1a] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] [1002:6819]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
06:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
07:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
07:02.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

I think 05:00.0 to 09:00.0 would be the combo card, but all 3 SATA Controller have [1b21:0612] vendor/device id. Any ideas what would be the best way to pass this whole card to the virtual machine? Do I need to pass those PCI Bridges, too?

Thanks in advance
Frank

cjn

Member

Registered: 2013-10-20

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I got hold of another Radeon HD 5450. I'm trying to pass it through to a Windows 7 guest while using an identical graphics card on the host, which is now running the radeon frame buffer driver (but no X). I still don't have much luck. In fact, the host crashes hard as soon as I start QEMU, right when the guest is loading the Windows 7 installer and the guest graphics card says "Windows is loading files...".

This is the relevant snippet from syslog when the crash happens

Nov  2 15:32:20 host kernel: [  746.027441] VFIO - User Level meta-driver version: 0.3
Nov  2 15:35:04 host kernel: [  911.086167] pcieport 0000:00:0d.0: driver skip pci_set_master, fix it!
Nov  2 15:35:04 host kernel: [  911.086173] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
Nov  2 15:35:07 host acpid: input device has been disconnected, fd 7
Nov  2 15:35:07 host kernel: [  914.162057] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none
Nov  2 15:35:08 host kernel: [  914.777896] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci
Nov  2 15:35:08 host kernel: [  915.138530] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci
Nov  2 15:35:09 host kernel: [  915.571288] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci
Nov  2 15:35:09 host kernel: [  915.935931] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci

and this is the VFIO debug output (warning: 4.5MB of logs). Not all of the relevant kernel messages are syslogged, unfortunately. Some only reach the console. I've seen crashes in both SATA and USB code.

My configuration is currently

Linux 3.12.0-rc7
QEMU head as of today (includes SeaBIOS) + nosnoop patch + debug-vfio patch
ASRock 990FX Extreme3
AMD FX-8350
Radeon HD 5450 (host, PCI address 01:00)
Radeon HD 5450 (guest, PCI address 05:00)

I start QEMU like this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/local/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/shm/windows.img,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-drive file=/root/en_windows_7_home_premium_with_sp1_x64_dvd_u_676549.iso,id=isocd \
-device ide-cd,bus=ahci.1,drive=isocd \
-nographic -usb -usbdevice host:04fe:0006 -usbdevice host:046d:c05b

The passed USB devices are my keyboard and my mouse.

Due to the fact that my two graphics cards have identical PCI IDs

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

and pci-stub accepts only PCI IDs as parameters (and not PCI addresses), I have

# Bind both graphics cards to the pci-stub driver and then unbind the one for
# the host, making sure that the pci-stub driver is loaded before the drm
# driver (and, in turn, before the radeon driver).
options pci-stub ids=1002:68e1,1002:aa68
install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:01:00.0 0000:01:00.1
softdep drm pre: pci-stub

in /etc/modprobe.d/pci-stub.conf and

#!/bin/sh
for dev in "$@"
do
  echo $dev > /sys/bus/pci/drivers/pci-stub/unbind
done

in /usr/local/sbin/pci-stub-unbind, to bind the guest graphics card to the pci-stub driver and the host graphics card to the radeon driver.

Anyway, how do I get rid of the crashes? They seem to be caused by the radeon driver because I get no crashes without it.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I got hold of another Radeon HD 5450. I'm trying to pass it through to a Windows 7 guest while using an identical graphics card on the host, which is now running the radeon frame buffer driver (but no X). I still don't have much luck. In fact, the host crashes hard as soon as I start QEMU, right when the guest is loading the Windows 7 installer and the guest graphics card says "Windows is loading files...".

This is the relevant snippet from syslog when the crash happens

Nov  2 15:32:20 host kernel: [  746.027441] VFIO - User Level meta-driver version: 0.3
Nov  2 15:35:04 host kernel: [  911.086167] pcieport 0000:00:0d.0: driver skip pci_set_master, fix it!
Nov  2 15:35:04 host kernel: [  911.086173] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
Nov  2 15:35:07 host acpid: input device has been disconnected, fd 7
Nov  2 15:35:07 host kernel: [  914.162057] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none
Nov  2 15:35:08 host kernel: [  914.777896] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci
Nov  2 15:35:08 host kernel: [  915.138530] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci
Nov  2 15:35:09 host kernel: [  915.571288] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci
Nov  2 15:35:09 host kernel: [  915.935931] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci

and this is the VFIO debug output (warning: 4.5MB of logs). Not all of the relevant kernel messages are syslogged, unfortunately. Some only reach the console. I've seen crashes in both SATA and USB code.

My configuration is currently

Linux 3.12.0-rc7
QEMU head as of today (includes SeaBIOS) + nosnoop patch + debug-vfio patch
ASRock 990FX Extreme3
AMD FX-8350
Radeon HD 5450 (host, PCI address 01:00)
Radeon HD 5450 (guest, PCI address 05:00)

I start QEMU like this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/local/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/shm/windows.img,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-drive file=/root/en_windows_7_home_premium_with_sp1_x64_dvd_u_676549.iso,id=isocd \
-device ide-cd,bus=ahci.1,drive=isocd \
-nographic -usb -usbdevice host:04fe:0006 -usbdevice host:046d:c05b

The passed USB devices are my keyboard and my mouse.

Due to the fact that my two graphics cards have identical PCI IDs

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

and pci-stub accepts only PCI IDs as parameters (and not PCI addresses), I have

# Bind both graphics cards to the pci-stub driver and then unbind the one for
# the host, making sure that the pci-stub driver is loaded before the drm
# driver (and, in turn, before the radeon driver).
options pci-stub ids=1002:68e1,1002:aa68
install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:01:00.0 0000:01:00.1
softdep drm pre: pci-stub

in /etc/modprobe.d/pci-stub.conf and

#!/bin/sh
for dev in "$@"
do
  echo $dev > /sys/bus/pci/drivers/pci-stub/unbind
done

in /usr/local/sbin/pci-stub-unbind, to bind the guest graphics card to the pci-stub driver and the host graphics card to the radeon driver.

Anyway, how do I get rid of the crashes? They seem to be caused by the radeon driver because I get no crashes without it.

You're supposed to bind the card to vfio-pci not pci-stub

cjn

Member

Registered: 2013-10-20

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You're supposed to bind the card to vfio-pci not pci-stub

Thanks, but I already do that, using the vfio-bind script. I run

vfio-bind 0000:05:00.0 0000:05:00.1

right before starting QEMU. This should unbind the graphics card from the pci-stub driver and bind it to the vfio-pci driver.

Also, my IOMMU group looks fine, as far as I can see

root@host:~# ll /sys/bus/pci/devices/0000\:05\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Nov  3 13:37 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:0d.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Nov  3 13:37 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:0d.0/0000:05:00.1

Have you had any luck passing through your Radeon HD 5450 (with the radeon driver loaded)? Or have you only used it on the host?

Norcoen

Member

From: Norway

Registered: 2013-11-02

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I Have a problem passing through my PCIe Combo Card to the virtual machine, because it has the same ASM1062 Serial ATA Controller as I have onboard.

# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5)
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 [8086:8c1a] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] [1002:6819]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
06:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
07:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
07:02.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa)
08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

I think 05:00.0 to 09:00.0 would be the combo card, but all 3 SATA Controller have [1b21:0612] vendor/device id. Any ideas what would be the best way to pass this whole card to the virtual machine? Do I need to pass those PCI Bridges, too?

Thanks in advance
Frank

I think I wrote this in a wrong way. I could pass them to the VM by using their enumeration, but how do I tell my kernel to ignore them on boot with pci-stub?

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"