KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 · 2013-07-26 20:51:40

nbhs wrote:

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
if you need the rom you can do:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
or using pci-assign instead of vfio

No idea how I didn't think of that yet, but it sadly doesn't seem to solve the problem. But why excactly should it work passed through, when the situation is like this: IOMMU in UEFI on → pata-marvell driver (on host) barely works, as in less than 4mb/s with hdparm or dd after a few minutes to recognize a device. IOMMU off, normal speeds, several 100mb/s from SSD.
Anyway, vfio with or without romfile and behind the pcie root port or not produce BSODs in windows 7+8 guests while loading the driver and pci-assign kind of crashes the host

Jul 26 22:36:58 desk kernel: ---[ end trace efd4627dfc7dbb30 ]---
Jul 26 22:36:58 desk kernel:  RSP <ffff8804475e1ce0>
Jul 26 22:36:58 desk kernel: RIP  [<ffffffff81141c64>] free_huge_page+0x124/0x140
Jul 26 22:36:58 desk kernel: Code: 10 00 00 00 ad de 48 89 43 20 48 b8 00 02 20 00 00 00 ad de 48 89 43 28 e8 ca e9 ff ff 49 ff 4c 24 38 41 ff 4c 24 70 eb 8f 0f 0b <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 66 66 2e 0f 
Jul 26 22:36:58 desk kernel:  [<ffffffff81621986>] system_call_fastpath+0x1a/0x1f
Jul 26 22:36:58 desk kernel:  [<ffffffff8161d82e>] ? do_page_fault+0xe/0x20
Jul 26 22:36:58 desk kernel:  [<ffffffff81174cc1>] SyS_ioctl+0x81/0xa0
Jul 26 22:36:58 desk kernel:  [<ffffffff8112f4d8>] ? do_munmap+0x2b8/0x3e0
Jul 26 22:36:58 desk kernel:  [<ffffffff81174a5d>] do_vfs_ioctl+0x2dd/0x4c0
Jul 26 22:36:58 desk kernel:  [<ffffffff814b3f38>] vfio_fops_unl_ioctl+0x78/0x360
Jul 26 22:36:58 desk kernel:  [<ffffffff8161d544>] ? __do_page_fault+0x2e4/0x5c0
Jul 26 22:36:58 desk kernel:  [<ffffffff814b5c31>] vfio_iommu_type1_ioctl+0x4b1/0x7a0
Jul 26 22:36:58 desk kernel:  [<ffffffff814b52ee>] vfio_dma_unmap+0xe/0x20
Jul 26 22:36:58 desk kernel:  [<ffffffff814b51bf>] __vfio_dma_do_unmap.isra.3+0x7f/0xa0
Jul 26 22:36:58 desk kernel:  [<ffffffff814b511a>] put_pfn+0x3a/0x60
Jul 26 22:36:58 desk kernel:  [<ffffffff8110e10c>] put_page+0x2c/0x40
Jul 26 22:36:58 desk kernel:  [<ffffffff8110df15>] put_compound_page+0x55/0x220
Jul 26 22:36:58 desk kernel:  [<ffffffff8110de5f>] __put_compound_page+0x1f/0x40
Jul 26 22:36:58 desk kernel: Call Trace:
Jul 26 22:36:58 desk kernel:  ffff880445b88428 0000000000000000 000000000f400000 0000000000000002
Jul 26 22:36:58 desk kernel:  ffffffff8110de5f ffffea000f400000 ffff8804475e1d58 ffffffff8110df15
Jul 26 22:36:58 desk kernel:  ffffea000f000000 ffffea000f000000 ffffea000f00001c ffff8804475e1d10
Jul 26 22:36:58 desk kernel: Stack:
Jul 26 22:36:58 desk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 26 22:36:58 desk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 26 22:36:58 desk kernel: CR2: 00007f509648a180 CR3: 000000033e0b7000 CR4: 00000000000407f0
Jul 26 22:36:58 desk kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 26 22:36:58 desk kernel: FS:  00007f509617d900(0000) GS:ffff88045ec00000(0000) knlGS:0000000000000000
Jul 26 22:36:58 desk kernel: R13: 0000000000000000 R14: 0000000000000246 R15: ffff8804475e1fd8
Jul 26 22:36:58 desk kernel: R10: 0000000000000000 R11: 000ffffffffff000 R12: ffffffff81ce3e20
Jul 26 22:36:58 desk kernel: RBP: ffff8804475e1cf8 R08: 0000000000000001 R09: ffff88044892b2a0
Jul 26 22:36:58 desk kernel: RDX: 800000000000001c RSI: 0000000000000003 RDI: 0000000040000000
Jul 26 22:36:58 desk kernel: RAX: 0000000000000000 RBX: ffffea000f000000 RCX: 0000000000000012
Jul 26 22:36:58 desk kernel: RSP: 0018:ffff8804475e1ce0  EFLAGS: 00010213
Jul 26 22:36:58 desk kernel: RIP: 0010:[<ffffffff81141c64>]  [<ffffffff81141c64>] free_huge_page+0x124/0x140
Jul 26 22:36:58 desk kernel: task: ffff880443d85040 ti: ffff8804475e0000 task.ti: ffff8804475e0000
Jul 26 22:36:58 desk kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./990FX Extreme4, BIOS P2.00 10/15/2012
Jul 26 22:36:58 desk kernel: CPU: 0 PID: 947 Comm: qemu-system-x86 Tainted: P           O 3.10.3-1-ck #1
Jul 26 22:36:58 desk kernel:  usbhid hid ata_generic ahci pata_acpi libahci ohci_hcd ehci_pci xhci_hcd ehci_hcd libata usbcore usb_common scsi_mod
Jul 26 22:36:58 desk kernel: Modules linked in: tun nfsd auth_rpcgss oid_registry nfs_acl bridge stp llc ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_na
Jul 26 22:36:58 desk kernel: invalid opcode: 0000 [#1] PREEMPT SMP 
Jul 26 22:36:58 desk kernel: kernel BUG at mm/hugetlb.c:627!
Jul 26 22:36:58 desk kernel: ------------[ cut here ]------------

and qemu says something about wrong PCI something size, not a multiple of 4k? If this might be relevant, I'll reproduce it and copy&paste the output.
It's been a while, so is this the right syntax for pci-assign?

-device pci-assign,host=02:00.0,romfile=/root/1b4b-9120_v1001033.bin,multifunction=on \
-device pci-assign,host=02:00.1,rombar=0

I still think it's a hardware error on Marvell's side that might be worked arround in the amd_iommu driver, but it would be nice if there was another way to solve this.

teekay · 2013-07-26 21:05:14

nbhs wrote:

andy123 wrote:
nbhs wrote:
Have you tried appending iommu=pt to your grub cfg?
syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]
The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.
Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
if you need the rom you can do:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
or using pci-assign instead of vfio

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Last edited by teekay (2013-07-26 21:08:44)

nbhs · 2013-07-26 21:07:06

teekay wrote:

nbhs wrote:
andy123 wrote:
syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]
The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.
Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
if you need the rom you can do:
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \
-device vfio-pci,host=02:00.1,bus=root.2,addr=00.1
or using pci-assign instead of vfio
The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

andy123 · 2013-07-26 21:16:01

nbhs wrote:

teekay wrote:
The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.
Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

teekay · 2013-07-26 21:24:39

andy123 wrote:

nbhs wrote:
teekay wrote:
The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.
Yeah that another thing you can try, binding it to pci-stub before the driver grabs it
Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?
p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

In the qemu command I use

-device vfio-pci,host=03:00.0,bus=pcie.0

and it works fine, no error messages on the host side. But as said, I needed to bind it to pci-stub, otherwhise the host's ahci driver would go nuts (but that's fully unrelated to qemu, passthrough and all that).

nbhs · 2013-07-26 21:31:26

andy123 wrote:

nbhs wrote:
teekay wrote:
The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.
Yeah that another thing you can try, binding it to pci-stub before the driver grabs it
Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?
p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

teekay · 2013-07-26 21:41:35

nbhs wrote:

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

Good point, nbhs is correct: it's the ahci driver. The pata-marvell is unrelated, so blacklisting it won't help. That's why only pci-stub can stop the host from acting upon the controller.

└» zcat /proc/config.gz |grep PATA_MARVELL
# CONFIG_PATA_MARVELL is not set

andy123 · 2013-07-26 21:42:23

nbhs wrote:

andy123 wrote:
nbhs wrote:
Yeah that another thing you can try, binding it to pci-stub before the driver grabs it
Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?
p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".
Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

That's what I though at first, but "lspci -nnvvv" tells me this:

02:00.0 IDE interface [0101]: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller [1b4b:91a0] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: ASRock Incorporation Device [1849:91a0]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 44
        Region 0: I/O ports at d090 [size=8]
        Region 1: I/O ports at d080 [size=4]
        Region 2: I/O ports at d070 [size=8]
        Region 3: I/O ports at d060 [size=4]
        Region 4: I/O ports at d050 [size=16]
        Region 5: Memory at fe615000 (32-bit, non-prefetchable) [size=2K]
        Expansion ROM at fe600000 [disabled] [size=64K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: pata_marvell, pata_acpi, ata_generic

02:00.1 IDE interface [0101]: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 45
        Region 0: I/O ports at d040 [size=8]
        Region 1: I/O ports at d030 [size=4]
        Region 2: I/O ports at d020 [size=8]
        Region 3: I/O ports at d010 [size=4]
        Region 4: I/O ports at d000 [size=16]
        Region 5: Memory at fe614000 (32-bit, non-prefetchable) [size=16]
        Expansion ROM at fe610000 [disabled] [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: pata_marvell, pata_acpi, ata_generic

And I found this patch on the kernel mailinglist, that seems to be responsible for this, but as said it works well without IOMMU.
And yes, this is indepented of AHCI or IDE setting in the UEFI. But iirc, it's always detected as an IDE controller in windows (bare-metal), too.
There are some quite strange controllers on this board, the USB3 is from Etron, it works better than the Marvell SATA controller, but sometimes freezes and caused BSODs in a guest, that's why I pass though one of the usb2 controllers now.

edit:

teekay wrote:

Good point, nbhs is correct: it's the ahci driver. The pata-marvell is unrelated, so blacklisting it won't help. That's why only pci-stub can stop the host from acting upon the controller.

Interpret the lspci output above for yourself, but I read from it, that I blacklisted the right driver. Please tell me if I didn't.

Last edited by andy123 (2013-07-26 21:48:49)

nbhs · 2013-07-26 21:45:59

andy123 wrote:

nbhs wrote:
andy123 wrote:
Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?
p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".
Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?
That's what I though at first, but "lspci -nnvvv" tells me this:
02:00.0 IDE interface [0101]: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller [1b4b:91a0] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: ASRock Incorporation Device [1849:91a0]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 44
        Region 0: I/O ports at d090 [size=8]
        Region 1: I/O ports at d080 [size=4]
        Region 2: I/O ports at d070 [size=8]
        Region 3: I/O ports at d060 [size=4]
        Region 4: I/O ports at d050 [size=16]
        Region 5: Memory at fe615000 (32-bit, non-prefetchable) [size=2K]
        Expansion ROM at fe600000 [disabled] [size=64K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: pata_marvell, pata_acpi, ata_generic

02:00.1 IDE interface [0101]: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 45
        Region 0: I/O ports at d040 [size=8]
        Region 1: I/O ports at d030 [size=4]
        Region 2: I/O ports at d020 [size=8]
        Region 3: I/O ports at d010 [size=4]
        Region 4: I/O ports at d000 [size=16]
        Region 5: Memory at fe614000 (32-bit, non-prefetchable) [size=16]
        Expansion ROM at fe610000 [disabled] [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: pata_marvell, pata_acpi, ata_generic
And I found this patch on the kernel mailinglist, that seems to be responsible for this, but as said it works well without IOMMU.
And yes, this is indepented of AHCI oder IDE setting in the UEFI. But iirc, it's always detected as an IDE controller in windows (bare-metal), too.
There are some quite strange controllers on this board, the USB3 is from Etron, it works better than the Marvell SATA controller, but sometimes freezes and caused BSODs in a guest, that's why I pass though one of the usb2 controllers now.

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

andy123 · 2013-07-26 21:48:21

nbhs wrote:

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

To quote myself:

andy123 wrote:

And yes, this is indepented of AHCI or IDE setting in the UEFI.

Last edited by andy123 (2013-07-26 21:48:34)

nbhs · 2013-07-26 21:55:02

andy123 wrote:

nbhs wrote:
Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci
To quote myself:
andy123 wrote:
And yes, this is indepented of AHCI or IDE setting in the UEFI.

Im sorry i overlooked that, does this behavior happens when you disable iommu?

nbhs · 2013-07-26 21:59:39

nbhs wrote:

andy123 wrote:
nbhs wrote:
Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci
To quote myself:
andy123 wrote:
And yes, this is indepented of AHCI or IDE setting in the UEFI.
Im sorry i overlooked that, does this behavior happens when you disable iommu?

EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59

Last edited by nbhs (2013-07-26 22:07:21)

andy123 · 2013-07-26 22:14:57

nbhs wrote:

nbhs wrote:
andy123 wrote:
To quote myself:
Im sorry i overlooked that, does this behavior happens when you disable iommu?
EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59

No Problem. I have this guide as paper, but it's kind of worthless, since it depicts the old, 1.x version of the UEFI, so wrong defaults and stuff.
As stated somewhere above, with IOMMU off, the driver works good (normal SSD speeds and devices recognized normally).

nbhs · 2013-07-26 22:18:39

andy123 wrote:

nbhs wrote:
nbhs wrote:
Im sorry i overlooked that, does this behavior happens when you disable iommu?
EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59
No Problem. I have this guide as paper, but it's kind of worthless, since it depicts the old, 1.x version of the UEFI, so wrong defaults and stuff.
As stated somewhere above, with IOMMU off, the driver works good (normal SSD speeds and devices recognized normally).

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

EDIT: something to read if you're interested http://www.projectosx.com/forum/index.p … topic=1150
http://wiki.debian.org/InstallingDebian … acBook/3-1

Last edited by nbhs (2013-07-26 22:27:08)

andy123 · 2013-07-26 22:27:04

nbhs wrote:

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

nbhs · 2013-07-26 22:35:51

andy123 wrote:

nbhs wrote:
Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours
I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci

parm:           marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

Last edited by nbhs (2013-07-26 23:09:11)

GizmoChicken · 2013-07-26 23:19:10

teekay wrote:

The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

nbhs wrote:

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

@teekay and nbhs:

Thanks much! I think I'll give Gigabyte a try next time.

nbhs wrote:

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:
vfio_iommu_type1.allow_unsafe_interrupts=1

@nbhs:

Thanks for this too. As you suggested, I passed "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" as kernel parameters, and now I no longer need the allow_unsafe_interrupts module. Great!

nbhs wrote:

Have you tried appending iommu=pt to your grub cfg?

I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does. Is there simple explanation of what this parameter does?

nbhs · 2013-07-26 23:25:55

GizmoChicken wrote:

teekay wrote:
The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.
nbhs wrote:
Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte
@teekay and nbhs:
Thanks much! I think I'll give Gigabyte a try next time.
nbhs wrote:
yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:
vfio_iommu_type1.allow_unsafe_interrupts=1
@nbhs:
Thanks for this too. As you suggested, I passed "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" as kernel parameters, and now I no longer need the allow_unsafe_interrupts module. Great!
nbhs wrote:
Have you tried appending iommu=pt to your grub cfg?
I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does. Is there simple explanation of what this parameter does?

I believe this disables DMA remapping

GizmoChicken · 2013-07-26 23:33:10

nbhs wrote:

GizmoChicken wrote:
I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does. Is there simple explanation of what this parameter does?
I believe this disables DMA remapping

Thanks!

GizmoChicken · 2013-07-27 06:44:34

IT WORKS!

For me, the trick for ending the BSODs that I previously reported was to replace the ahci controller ("-device ahci,bus=pcie.0,id=ahci") with an ide controller ("-device piix4-ide,bus=pcie.0"). It took me a few tries to figure out that the change in controller necessitated replacing "-device ide-hd,bus=ahci.0,drive=disk" with "-device ide-hd,drive=disk" (no "bus=...,") compared to the original example.

Here's a minimalist version of the qemu commands that I'm using:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
-device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root,addr=00.1 \
-device piix4-ide,bus=pcie.0 \
-drive file=/images/Win7HVM.img,id=disk,format=raw -device ide-hd,drive=disk

Even with an outdated Nvidia driver (downloaded with Windows updates), I have HDMI audio and a Windows experience of 6.9 for desktop graphics and gaming graphics. But something tells me that a newer driver would improve matters.

Here's a summary of my setup:

MB:  ASUS M5A99FX PRO R2.0 w/ BIOS v2005
CPU:  AMD Phenom II X4 955
RAM: 20 GB DDR3 1600
Primary GPU: Radeon HD6670 w/ a Catalyst 13.6-beta driver on the host
Seconday GPU: Nvidia GTX550Ti w/ an Nvidia driver (from Windows updates) on the guest
Ubuntu 13.10 (current daily build)
Kernel 3.10.0-5  (from Ubuntu, no patches)
seabios 1.7.3  (from Ununtu, no patches)
qemu 1.5.0 (from Ununtu, no patches)

Since I'm using a stock Ubuntu kernel, I added the following to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd
amd_iommu_v2
pci_stub

Also, here's the GRUB cmdline that I'm using:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci-stub.ids=10de:1244,10de:0bee"

Using BIOS v2005, IOMMU works flawlessly on my ASUS M5A99FX PRO R2.0 motherboard. Using older BIOS versions with that motherboard, IOMMU errors occurred, but those errors could be cured by adding "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" to the GRUB cmdline (as suggested by nbhs). (Note that the pci addresses and values "[5]" and "[6]" may need to be changed, depending on setup.)

Using BIOS v2005, "dmesg | grep AMD-Vi" gives me the following output:

$ dmesg | grep AMD-Vi
AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Interrupt remapping enabled
AMD-Vi: Lazy IO/TLB flushing enabled

I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU. (I never was able to get my Ubuntu guest to boot with vfio passthough.)

Now, if only I could use VirtManager to start/stop/monitor this VM.

EDIT 1: I made a few minor tweaks since I first posted the above. For details, have a look here: https://bbs.archlinux.org/viewtopic.php … 7#p1313007
EDIT 2: Updated to reflect that BIOS v2005 cures previously reported IOMMU errors and other small tweaks.

Last edited by GizmoChicken (2013-11-08 04:03:16)

GizmoChicken · 2013-07-27 08:09:02

GizmoChicken wrote:

I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU. (I never was able to get my Ubuntu guest to boot with vfio passthough.)

Small update: I just noticed that, when booting from an iso and running in "try it" mode, I am able to passthough an Ubuntu 13.10 guest to the GPU. So my guess is that vfio doesn't like something about the Ubuntu VM that I had previously created using VirtManager. At some point, I'll try creating a fresh Ubuntu VM and test again.

nbhs · 2013-07-27 08:58:23

GizmoChicken wrote:

IT WORKS!
For me, the trick for ending the BSODs that I previously reported was to replace the ahci controller ("-device ahci,bus=pcie.0,id=ahci") with an ide controller ("-device piix4-ide,bus=pcie.0"). It took me a few tries to figure out that the change in controller necessitated replacing "-device ide-hd,bus=ahci.0,drive=disk" with "-device ide-hd,drive=disk" (no "bus=...,") compared to the original example.
Here's a minimalist version of the qemu commands that I'm using:
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
-device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root,addr=00.1 \
-device piix4-ide,bus=pcie.0 \
-drive file=/images/Win7HVM.img,id=disk,format=raw -device ide-hd,drive=disk
Even with an outdated Nvidia driver (downloaded with Windows updates), I have HDMI audio and a Windows experience of 6.9 for desktop graphics and gaming graphics. But something tells me that a newer driver would improve matters.
Here's a summary of my setup:
MB:  ASUS M5A99FX PRO R2.0
CPU:  AMD Phenom II X4 955
RAM: 20 GB DDR3 1600
Primary GPU: Radeon HD6670 w/ a Catalyst 13.6-beta driver on the host
Seconday GPU: Nvidia GTX550Ti w/ an Nvidia driver (from Windows updates) on the guest
Ubuntu 13.10 (current daily build)
Kernel 3.10.0-5  (from Ubuntu, no patches)
qemu 1.5.0 (from Ununtu, no patches)
seabios 1.7.3  (from Ununtu, no patches)
Since I'm using a stock Ubuntu kernel, I added the following to /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd
amd_iommu_v2
pci_stub
Also, here's the GRUB cmdline that I'm using:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 pci-stub.ids=10de:1244,10de:0bee"
As I mentioned elsewhere, the addition of "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" (as suggested by nbhs) seems to cure the IOMMU errors with ASUS M5A99FX PRO R2.0 motherboard. (Note that the pci addresses and values "[5]" and "[6]" may need to be changed, depending on setup.)
$ dmesg | grep AMD-Vi
AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Interrupt remapping enabled
AMD-Vi: Lazy IO/TLB flushing enabled
I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU. (I never was able to get my Ubuntu guest to boot with vfio passthough.)
Now, if only I could use VirtManager to start/stop/monitor this VM.

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Last edited by nbhs (2013-07-27 08:59:17)

GizmoChicken · 2013-07-27 10:35:12

nbhs wrote:

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Yep, that probably explains it. I'll create fresh VMs (Windows and Ubuntu) using sata/ahci next week and will test whether they too run with the unpatched versions of qemu and seabios along with the stock Ubuntu 3.10 kernel.

Oh, I tested my Nvidia GT610 as yet another secondary GPU, and it seems to be working great too. Not quite as fast as my Nvidia GTX550Ti, but plenty fast for Netflix and Youtube.

QUESTION: Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent. And so I'm thinking that I might like a matching GT610 for use as a primary card on my host. But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card? That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

Last edited by GizmoChicken (2013-07-27 10:42:05)

myweb · 2013-07-27 11:35:39

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

Last edited by myweb (2013-07-27 11:37:51)

myweb · 2013-07-27 13:46:47

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Arch Linux

#276 2013-07-26 20:51:40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#277 2013-07-26 21:05:14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#278 2013-07-26 21:07:06

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#279 2013-07-26 21:16:01

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#280 2013-07-26 21:24:39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#281 2013-07-26 21:31:26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#282 2013-07-26 21:41:35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#283 2013-07-26 21:42:23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#284 2013-07-26 21:45:59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#285 2013-07-26 21:48:21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#286 2013-07-26 21:55:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#287 2013-07-26 21:59:39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#288 2013-07-26 22:14:57

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#289 2013-07-26 22:18:39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#290 2013-07-26 22:27:04

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#291 2013-07-26 22:35:51

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#292 2013-07-26 23:19:10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#293 2013-07-26 23:25:55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#294 2013-07-26 23:33:10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#295 2013-07-27 06:44:34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#296 2013-07-27 08:09:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#297 2013-07-27 08:58:23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#298 2013-07-27 10:35:12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#299 2013-07-27 11:35:39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#300 2013-07-27 13:46:47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Board footer