Arch Linux

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/run/media/user/CopiedData - is a hard drive with NTFS filesystem (/ is placed on SSD, soft archive, media, etc on usual hard drive)
I have tried vice versa: place installation image on SSD and windows to hard drive - 10 minutes passed and partitioning window is shown.
I also tried place windows to SSD and installation image to RAM disk - result is the same - freeze.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/run/media/user/CopiedData - is a hard drive with NTFS filesystem (/ is placed on SSD, soft archive, media, etc on usual hard drive)
I have tried vice versa: place installation image on SSD and windows to hard drive - 10 minutes passed and partitioning window is shown.
I also tried place windows to SSD and installation image to RAM disk - result is the same - freeze.

Does this also happen using the the ide or virtio controller instead of ahci?

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried to execute with virtio:

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

windows installer does not recognize any hard drive

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried to execute with virtio:

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

windows installer does not recognize any hard drive

Try the ide controller, its on the guide, im really not sure whats causing your problem, i think another user had the same problem a few pages back, if that doesnt work try using the old chipset (remove the -M q35 part)

Last edited by nbhs (2013-07-18 21:39:47)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried:
qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd \
-drive file=/home/abrilevskiy/windows.img,if=ide,format=raw,index=2

but VM could not boot: boot drive is not found

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It looks like I mistaken with parameters.

qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

but I receive: qemu-system-x86_64: -device ide-cd,bus=ide.1,drive=isocd: Property 'ide-cd.drive' can't take value 'isocd', it's in use

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried:
qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd \
-drive file=/home/abrilevskiy/windows.img,if=ide,format=raw,index=2

but VM could not boot: boot drive is not found

qemu-system-x86_64 -m 1024 -M q35 -enable-kvm -boot menu=on -cpu host  -device piix4-ide,bus=pcie.0 -drive file=/home/nbhs/windows.img,id=windisk,format=raw -device ide-hd,bus=ide.0,drive=windisk -drive file=/home/nbhs/windows8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

works for me

EDIT:

qemu-system-x86_64 -m 1024 -enable-kvm -boot menu=on -cpu host -hda /home/nbhs/windows.img -cdrom windows8.iso

also works (-hda and -cdrom wont work using the q35 chipset), to use virtio i think all you need to do is add if=virtio at the end and it will automatically add the virtio controller too., tho you'll probably need the drivers, ex:

qemu-system-x86_64 -m 1024 -enable-kvm -boot menu=on -cpu host -drive file=/home/nbhs/windows.img,id=windisk,format=raw,if=virtio -drive file=/home/nbhs/windows8.iso,id=isocd,if=virtio

Last edited by nbhs (2013-07-18 22:26:56)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -m 4096  -M q35 -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot menu=on \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

works for me, thank you!

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -m 4096  -M q35 -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot menu=on \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

works for me, thank you!

No problem once you install it you might want to use virtio drivers, or the ahci controller, see my edited post above.

Evonat

Member

Registered: 2013-07-24

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

Well it should be possible, i am able to passthrough my integrated sound card to a vm and it works fine

reacocard

Member

Registered: 2008-05-25

Posts: 20

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

You can probably pass-through the second card just fine like any other pci device. As far as the headphones go, if you want audio from both cards I'd suggest either just combining the outputs by cables with something like this http://amzn.com/B0016LDZ36 or connecting the line-out of the integrated audio to the line-in on the Xonar. The latter method has the downside that audio from the integrated card becomes dependent on the proper operation of the VM with the Xonar however.

GizmoChicken

Member

Registered: 2013-07-25

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

[...]

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work

@nbhs:

This is a GREAT tutorial.  Thanks so much for creating it!

Because I'm not quite ready to invest the time into applying the suggested patches, I thought I'd test my luck by trying this tutorial on a daily build of Ubuntu 13.10 that included the following, but without any patches:

Mainline Kernel 3.11 RC2
qemu 1.5.0
seabios 1.7.3

And I added the following to /etc/modules: 

vfio
vfio_iommu_type1
vfio_pci
pci_stub
kvm
kvm_amd

THE GOOD NEWS:  Even without the patches or custom kernel, the above suggested test was successful.  That is, I saw the black qemu window on my main display, and seabios ouput on my monitor from my passthru'd card saying it cant find anything to boot.  So far, so good.

Now to try booting a VM:

DISK:

Creating and passing a disk image to qemu:

By default -M q35 creates an emulated sata controller, but it seems its not working on windows yet, so we attach the old ahci controller:

-device ahci,bus=pcie.0,id=ahci

[...]

And attach a disk to it:

-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

I had previously created a Windows 7 HVM using Virt-Manager under KVM (but without vfio), and so I plugged into the above the appropriate path to the Windows 7 image.   Much to my delight, when I attempted to boot the HVM, the monitor from my passthru'd card displayed boot information after which the Windows 7 welcome screen appeared for a few seconds.   But just when I was about to celebrate.... BSOD.

So I'm wondering:

(1)  Am I mistaken to think that, barring other problems, the Windows 7 HVM that I previously created using Virt-Manager under KVM should boot?

(2) Provided that the Windows 7 HVM that I created as described above should boot, what is the likelihood that the recommended patches will cure the BSOD that I'm experiencing?  That is, were either of the patches created to address issues that arise late into the boot phase, like I'm experiencing?  Or were the patches created to address different issues?

UPDATE:  I managed to get this working.   Have a look HERE for details.

Last edited by GizmoChicken (2013-07-27 06:48:54)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

[...]

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work

@nbhs:

This is a GREAT tutorial.  Thanks so much for creating it!

Because I'm not quite ready to invest the time into applying the suggested patches, I thought I'd test my luck by trying this tutorial on a daily build of Ubuntu 13.10 that included the following, but without any patches:

Mainline Kernel 3.11 RC2
qemu 1.5.0
seabios 1.7.3

And I added the following to /etc/modules: 

vfio
vfio_iommu_type1
vfio_pci
pci_stub
kvm
kvm_amd

THE GOOD NEWS:  Even without the patches or custom kernel, the test was successful.  That is, I saw the black qemu window on my main display, and seabios ouput on my monitor from my passthru'd card saying it cant find anything to boot.  So far, so good.

Now to try booting a VM:

DISK:

Creating and passing a disk image to qemu:

By default -M q35 creates an emulated sata controller, but it seems its not working on windows yet, so we attach the old ahci controller:

-device ahci,bus=pcie.0,id=ahci

[...]

And attach a disk to it:

-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

I had previously created a Windows 7 HVM using Virt-Manager under KVM (but without vfio), and so I plugged into the above the appropriate path to the Windows 7 image.   Much to my delight, when I attempted to boot the HVM, the monitor from my passthru'd card displayed boot information after which the Windows 7 welcome screen appeared for a few seconds.   But just when I was about to celebrate.... BSOD.

So I'm wondering:

(1)  Am I mistaken to think that, barring other problems, the Windows 7 HVM that I previously created using Virt-Manager under KVM should boot?

(2) Provided that the Windows 7 HVM that I created as described above should boot, what is the likelihood that the recommended patches will cure the BSOD that I'm experiencing?  That is, were either of the patches created to address issues that arise late into the boot phase, like I'm experiencing?  Or were the patches created to address different issues?

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

GizmoChicken

Member

Registered: 2013-07-25

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

@nbhs:

Thanks much for the quick reply!

As for the BSOD, it appears and disappears very quickly, and so unfortunately, I haven't had a chance to read the details.  But I can tell you that it always occurs just about when I would expect to see the Windows 7 login prompt.

As for my current test system, it includes:

MB:  ASUS M5A99FX PRO R2.0 ("options vfio_iommu_type1 allow_unsafe_interrupts=1" applied to overcome IOMMU issue)
CPU:  AMD Phenom II X4 955  (I have access to a newer CPU, but this one is usually reliable and works well with Xen.)
RAM: 20 GB ddr3 1600
GPUs: I've tried various combinations of a Radeon HD6670, an Nvidia GT610, and an Nvidia GTX550Ti.

This GPU combination passed your initial test (no drive attached) and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination also passed your initial test  and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Radeon HD6670 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination passed your initial test, but the screen attached to the secondary GPU remained BLANK (without any BSOD) when attempting to boot from an image:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Radeon HD6670 <-- Windows display

I'm surprised that the monitor attached to the Radeon HD6670 remained blank when attempting to boot from an image.  My Radeon HD6670 consistently works well with GPU passthrough under Xen, whereas GPU passthough has never worked with either of my Nvidia cards under Xen.

When I get a chance, I'll try with the patches applied to qemu and seabios.   

REQUEST:  I'm a Linux neophyte.  Would you be so kind as to suggest a tutorial on how to apply patches?

Last edited by GizmoChicken (2013-07-26 02:30:37)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

@nbhs:

Thanks much for the quick reply!

As for the BSOD, it appears and disappears very quickly, and so unfortunately, I haven't had a chance to read the details.  But I can tell you that it always occurs just about when I would expect to see the Windows 7 login prompt.

As for my current test system, it includes:

MB:  ASUS M5A99FX PRO R2.0 ("options vfio_iommu_type1 allow_unsafe_interrupts=1" applied to overcome IOMMU issue)
CPU:  AMD Phenom II X4 955  (I have access to a newer CPU, but this one is usually reliable and works well with Xen.)
RAM: 20 GB ddr3 1600
GPUs: I've tried various combinations of a Radeon HD6670, an Nvidia GT610, and an Nvidia GTX550Ti.

This GPU combination passed your initial test (no drive attached) and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination also passed your initial test  and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Radeon HD6670 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination passed your initial test, but the screen attached to the secondary GPU remained BLANK (without any BSOD) when attempting to boot from an image:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Radeon HD6670 <-- Windows display

I'm surprised that the monitor attached to the Radeon HD6670 remained blank when attempting to boot from an image.  My Radeon HD6670 consistently works well with GPU passthrough under Xen, whereas GPU passthough has never worked with either of my Nvidia cards under Xen.

When I get a chance, I'll try with the patches applied to qemu and seabios.   

REQUEST:  I'm a Linux neophyte.  Would you be so kind as to suggest a tutorial on how to apply patches?

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch

Last edited by nbhs (2013-07-26 03:03:39)

GizmoChicken

Member

Registered: 2013-07-25

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch

Thanks much.  When I get a chance, I'll create a clean Windows VM.  And I might even bite the bullet and try the patches.  But since I want to stick with Ubuntu and like to keep my system up-to-date, I'm hoping that the patches will make their way upstream soon.

By the way, I saw that you reported the IOMMU problem with your motherboard to Asus.   A few of us got together and wrote to the same Asus customer rep who has volunteered to be a single point of contact for this issue.  If you'd like to write to that customer rep, please send me a pm and I'll provide his contact information.

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch

Thanks much.  When I get a chance, I'll create a clean Windows VM.  And I might even bite the bullet and try the patches.  But since I want to stick with Ubuntu and like to keep my system up-to-date, I'm hoping that the patches will make their way upstream soon.

By the way, I saw that you reported the IOMMU problem with your motherboard to Asus.   A few of us got together and wrote to the same Asus customer rep who has volunteered to be a single point of contact for this issue.  If you'd like to write to that customer rep, please send me a pm and I'll provide his contact information.

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

GizmoChicken

Member

Registered: 2013-07-25

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I saw that you reported the IOMMU problem with your motherboard to Asus....

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Yep, I saw your earlier post:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

Last edited by GizmoChicken (2013-07-26 09:58:41)

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

---

i'm sorry for my poor english wirting skills…

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I saw that you reported the IOMMU problem with your motherboard to Asus....

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Yep, I saw your earlier post:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

Last edited by nbhs (2013-07-26 19:16:12)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

Last edited by nbhs (2013-07-26 19:16:56)

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.

---

i'm sorry for my poor english wirting skills…

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

if you need the rom you can do:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

or using pci-assign instead of vfio

Last edited by nbhs (2013-07-26 19:54:51)