Arch Linux

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1

/init/ line 1 = ivrs_ioapic[9]=00:14.0  : Command not found
/init/ line 1 = ivrs_ioapic[10]=00:00.1   : Command not found

or something like that.

Wait, how could i miss that part of his message?>_<
Sorry for bothering guys.

@Zeorymer:
Are you sure you've put that command line switches into the right... place?
Cause if you're running kernel newer than 3.19(where there's a patch for that) - that line is your only solution apart from aw's ultimate one.

Kernel was posted several posts ago: 4.0.7
Processor is an 8350.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="iommu=debug,verbose amd_iommu_dump=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1"
GRUB_CMDLINE_LINUX=""
#

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable Hidden Menu, and optionally hide the timeout count
#GRUB_HIDDEN_TIMEOUT=5
#GRUB_HIDDEN_TIMEOUT_QUIET=true

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter 
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx" 
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper 
# modes only.  Entries specified as foreground/background.
GRUB_COLOR_NORMAL="light-blue/black"
GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

#GRUB_SAVEDEFAULT="true"

This is where I put it, is this wrong?  I know there are other things in there that I recently added to see if I can get other information (first one doesn't seem to do anything new).

What I find odd id that I get the command not found messages but I don't get the IVRS messages being mapped wrong...which leads me to believe that the kernel THINKS the command is being passed(?)

So...a new board is the only solution?  Damn you Asus.

/init/ line 1 = ivrs_ioapic[9]=00:14.0  : Command not found
/init/ line 1 = ivrs_ioapic[10]=00:00.1   : Command not found

Those lines mean nothing, i get them too, the kernel arguments get passed you can check by doing cat /proc/cmdline, it has something to do with the mknitcpio script parsing the kernel parameters, but its nothing to worry about

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I hit a brick wall trying to add SATA passthrough to my setup. Passing through the seconday onboard controller, a Marvell 6121 if I remember correctly, locked everything up, and I mean everything. Even the hardware reset button did nothing. WTF? Obviously I could not get any error messages out of this. I purchased a PCIe SATA addon card. That one's a Marvell 91xx. The thought of using something else than Marvell crossed my mind but I couldn't find anything reasonable at the moment.

I connected one of my SSDs to the device. My grub is actually on that SSD but the drive is not needed afterwards and my system boots fine with the controller pci-stubbed. Of course I wouldn't be able to install anything on it without breaking my host OS but this is just a proof of concept to see if it works before I rearrange things.

Now I don't get the hard crash but the SATA passthrough still does not work. For one thing, SeaBIOS takes long, maybe almost a minute before anything else than the SeaBIOS version string appears every time I pass through the devices on the addon card. When it gets going, I can't get any bootable devices, not even the virtualised ones defined with the -drive option. No install media or disk images show up. Enabling the boot menu shows only the virtualised floppy drive, PXE boot and such. I tried starting a very minimalistic setup to rule out everything else but the result is the same.

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1

where 7:00.0 is the SATA device and 7:00.1 is an IDE device on the same card. They share the same IOMMU group so I'm passing both of them. The install medium is an untouched Windows 7 SP1 image. I also tried installing the guest to a virtualised image and adding the SATA controller on the next boot. Everything gone again. I cannot understand why even the virtualised drives disappear. Am I doing something obviously wrong? As for providing debug output, I haven't found anything to provide. Nothing appears in dmesg when things go haywire. I initially got "nobody cared about IRQ" messages but moving the SATA card to another PCIe slot with its own IRQ fixed those. I see libvirt is the common way nowadays. I tried the SeaBIOS way with virt-manager but it behaved exactly the same as with using qemu directly.

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

i dont think OVMF is able to boot from a sata controller right now, i posted a patched binary like 50 pages back, theres also ovmf-fishman-git on aur

Last edited by nbhs (2015-07-10 21:34:49)

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

i dont think OVMF is able to boot from a sata controller right now, i posted a patched binary like 50 pages back, theres also ovmf-fishman-git on aur

Good to know, thank you. I also just noticed Alex (thank you as well for all the materials on VFIO and passthrough) has said Windows 8+ is needed for OVMF, my bad. From what I have read, the 64-bit version of 7 should work with UEFI so I thought I would be fine.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.

Off-topic:
#5555
Could you please explain what...
"made me back-burner it" meant? My foolish mind only finds associations with butthurt.
On-topic:
Win7 with OVMF requires some special hacks - you should add QXL graphics device for the duration of install, otherwise you won't be able to see anything.
This is covered in OVMF whitepaper.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

http://idioms.thefreedictionary.com/on+the+back+burner

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

http://idioms.thefreedictionary.com/on+the+back+burner

Well...
Seems like it's the different cultures - if not the same size, all the big burners on the stove are located farther away from the edge. I guess it has to do something with fire safety. And usually that burners are used for some long-term powerful cooking, like a 10L of some soup. So yeah, it kinda makes sense.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

"Removed"?
Correct me if i'm wrong, but check again, i think some of them were default on.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

"Removed"?
Correct me if i'm wrong, but check again, i think some of them were default on.

As in, I removed them.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Don't tell us what you did, show us

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I get home I'll upload the domain XML, is there anything else you need to see?

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OT

I guess it has to do something with fire safety. And usually that burners are used for some long-term powerful cooking, like a 10L of some soup. So yeah, it kinda makes sense.

Don't any of you cook? Or do you just sit at the computer all day until food magically appears courtesy of some long neglected spouse? :roll eyes:

Last edited by mostlyharmless (2015-07-11 14:41:44)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgraded to 4.2-rc1 :

All of the VMs with GPUs passed through to them suffer from no output on their respective screens , checking htop , these VMs are stuck on 100% CPU load on their threads .

dmesg shows no errors or warnings . QEMU command line doesn't return any errors or warnings either .

Reverting back to 4.1-rc7 , and everything is working like it should . (No config or script files changes were done , just upgrading & downgrading the kernel) .

Last edited by Denso (2015-07-11 22:59:26)

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success at last! Though not without lots of setbacks and cursing at my computer.

With OVMF and Windows 8 I managed to pass through my add-on Marvell controller. However, with the latest OVMF build I could not install Windows. At the point the installer starts copying files to the new system partition, I got a BSOD (or is it a frowny face these days?) with a KMODE_EXCEPTION_NOT_HANDLED error code. This only occurred when running on OVMF, not on SeaBIOS. In UEFI mode, Windows also seems to be VERY pissy about showing disk drives that are not exactly the way it wants (GPT and such), even with drive images that should be empty. I recall it was just as terrible when I tried without success to install Win 7 in UEFI mode on this hardware back in 2011 so no surprises there.

I dug up the SATA patched OVMF binaries as hinted by nbhs. With them, the installer finished without problems! I have not tried installing on a passed-through drive or booting from one yet. The system drive is on VirtIO and the drive on the passed controller is a secondary one that I added after installation. During the first boot with the controller passed through I got a BSOD related to viostor (VirtIO). It also froze my host, but at least the reset button still worked . Next boot went smoothly. Don't know what went wrong there.

Even if this is as far as I'll manage to get, it should already be helpful if the games are installed on the passed drive and load their stuff from there. On a side note, the Marvell controller's performance is shoddy. I get 180MB/90MB R/W sequential. Googling revealed that it seems to be on par with what can be expected. Access times are down to 0.3-0.5 ms though, which is nice. I have not tried games on the passed drive yet but I'm hoping the IO related stuttering isn't there anymore. Lots of other optimisations such as cpu pinning still to do, anyway.

sunmast

Member

Registered: 2014-12-16

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to run OSX with OVMF method?

Schlunze

Member

Registered: 2013-10-03

Posts: 53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.

Yes, like others here mentioned all disks reconized by win7uefi need gpt. For dual boot you need to add legacy ide drivers in win registry if virtio drivers are used in libvirt/qemu.
On my machine sata controller-passtrough never worked the expected way, also with marvell-sata-controller. Others got it running with Asrock Z97 Extreme6. One time i´ve seen the disk during win7 ovmf installation, after installation reboot, it also hung up. Maybe it was the graphic-card like mentioned.

I took the route without sata-controller passthrough since i´ve read in several posts, performance is comparable and system-stability without is better.
Don´t know exactly if this is right.

Good luck @Impulse_255

fld

Member

Registered: 2015-02-06

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Aww, going from Catalyst 14.12 to Catalyst 15.7 on Win7 wasn't as easy as I had hoped

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgraded to 4.2-rc1 :

All of the VMs with GPUs passed through to them suffer from no output on their respective screens , checking htop , these VMs are stuck on 100% CPU load on their threads .

dmesg shows no errors or warnings . QEMU command line doesn't return any errors or warnings either .

Reverting back to 4.1-rc7 , and everything is working like it should . (No config or script files changes were done , just upgrading & downgrading the kernel) .

Thanks for the report!  I've bisected it back to this commit.

In my testing, the VM is not hung, but it does take over 90 seconds to get the OVMF splash screen (Seabios is less affected but still noticeable).  I've reported the problem upstream.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Thanks ! Actually it takes around 5 minutes for me , and the VM operates in Slow motion mode . (When a key stroke is sent the actual response happens 5 seconds later) .

Hopefully it will be fixed for -rc2 or 3 .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Thanks ! Actually it takes around 5 minutes for me , and the VM operates in Slow motion mode . (When a key stroke is sent the actual response happens 5 seconds later) .

Hopefully it will be fixed for -rc2 or 3 .

The only issue I see is pre-boot, once the guest OS is running, performance is normal.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://lists.nongnu.org/archive/html/qe … 00021.html
Isn't that's just what VFIO was made for? I mean, Virtual Function...

ALSO:
https://support.microsoft.com/en-us/kb/314082/en-us
Finally found an article for those of you who experiment with various hard drive controllers, like virtio, bare metal and stuff.

Last edited by Duelist (2015-07-12 20:40:34)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So my VM is pretty stable and games are playable, however I'm having some serious problems with audio.

Passing through host audio with -soundhw hda gives me verybad DPC latency, spiking up to 8000 microseconds regularly and never drops below 1000.  Running the process using round robin scheduler as reccomended on page 6 works helps, DPC latency is stable at 1000 and never spiking however whenever I quit the VM, qemu process redlines all my processor cores and my computer becomes very unresponsive unless I am able to tab to a terminal and killall qemu-system-x86_64.

I also have a USB razer kraken headset, however passing it using -usbdevice there is much audio cracking and nothing I try reduces that at all.

I've tried using both alsa and pulse for audio and neither work better than the other... anyone have any suggestions on how I can reduce my DPC latency to normal values?

Edit:

Specifications:
Processor - Intel i5 4690k
Memory - 16GB
VM GPU - Radeon R9 290
Host GPU - Intel iGPU
Guest OS - Windows 8.1 64bit

qemu startup script:

#!/bin/bash

#Start synergy client
systemctl start synergyc@peetipablo

#Screen setup
xrandr --output DP1 --off
xrandr --output HDMI1 --off
xrandr --output HDMI3 --primary --auto
sleep 1

#start QEMU emulation
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa
qemu-system-x86_64 -name win8 -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Haswell,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
-bios OVMF.fd -m 8192 -mem-path /dev/hugepages -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 \
-localtime -nographic \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/OVMF.fd \
-drive if=pflash,format=raw,file=/usr/share/ovmf/OVMF_VARS-pure-efi.fd \
-device virtio-scsi-pci,id=scsi \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-drive file=/dev/mapper/sdb1_crypt,cache=none,id=hdd,format=raw,if=none -device scsi-hd,drive=hdd \
-drive file=/home/peetipablo/VMs/virtio-win-0.1.96.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-device virtio-net,netdev=net0,mac=DE:AD:BE:EF:EB:B7 -netdev tap,ifname=tap0,id=net0 \
-soundhw hda \
-vga none

#Restore screen
xrandr --output HDMI1 --off
xrandr --output DP1 --primary --mode 1920x1080 --rate 144 --left-of HDMI3
xrandr --output HDMI3 --auto --right-of DP1

#Stop synergy client
systemctl stop synergyc@peetipablo

Last edited by Peetipablo (2015-07-12 21:29:35)