Arch Linux

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I seem to be having the same issue that apoapo and myweb are running through.

- With the Radeon as the primary video device, I get apoapo's dirty stream of errors :

system-x86_64: vfio_bar_write(,0x13ff50, 0x36703067,4) failed: Device or resource busy

and this in dmesg:

[  763.408269] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]
[  763.408280] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]
[  763.408291] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

With Intel integrated as the primary device, no errors come up, but the Radeon never initializes.

The latter seems like an easier start point. Is there any way to force that card to turn on during the SEABios boot? Any way to log why it's not doing so?

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci

parm:           marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add  ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

I read that wikipedia article a while back, but it doesn't work. With ahci.marvell_enable=1 the driver even gets loaded automatically, but it doesn't recognize any devices (at least with iommu on, but the pata-marvell driver works decent with iommu off, so why try to force the ahci?). Relevant dmesg (dmesg|egrep -i (ata|ahci|scsi))

[    0.955526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.990232] SCSI subsystem initialized
[    0.993158] ACPI: bus type ATA registered
[    0.993369] libata version 3.00 loaded.
[    0.994387] pata_acpi 0000:02:00.1: enabling device (0000 -> 0003)
[    1.238497] ahci 0000:00:11.0: version 3.0
[    1.238729] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.238734] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
[    1.240134] scsi0 : ahci
[    1.240202] scsi1 : ahci
[    1.240277] scsi2 : ahci
[    1.240333] scsi3 : ahci
[    1.240418] scsi4 : ahci
[    1.240470] scsi5 : ahci
[    1.240513] ata1: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19
[    1.240515] ata2: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19
[    1.240517] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19
[    1.240519] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19
[    1.240521] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19
[    1.240523] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19
[    1.240619] ahci 0000:02:00.0: irq 75 for MSI/MSI-X
[    1.240632] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1
[    1.702088] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.702107] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.702123] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.703241] ata6.00: ATA-8: ST31000520AS, CC32, max UDMA/133
[    1.703243] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    1.704090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.704561] ata6.00: configured for UDMA/133
[    1.705090] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.705109] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.705619] ata1.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133
[    1.705620] ata1.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.706205] ata1.00: configured for UDMA/133
[    1.706285] scsi 0:0:0:0: Direct-Access     ATA      WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5
[    1.708208] ata3.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133
[    1.708210] ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.710489] ata2.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7
[    1.710491] ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.711470] ata4.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133
[    1.711472] ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.713723] ata5.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133
[    1.713725] ata5.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    1.714324] ata3.00: configured for UDMA/133
[    1.716961] ata2.00: configured for UDMA/133
[    1.717032] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD103SI  1AG0 PQ: 0 ANSI: 5
[    1.717202] scsi 2:0:0:0: Direct-Access     ATA      SAMSUNG HD204UI  1AQ1 PQ: 0 ANSI: 5
[    1.718470] ata4.00: configured for UDMA/133
[    1.718535] scsi 3:0:0:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[    1.723710] ata5.00: configured for UDMA/133
[    1.723771] scsi 4:0:0:0: Direct-Access     ATA      ADATA SP900      5.0. PQ: 0 ANSI: 5
[    1.723902] scsi 5:0:0:0: Direct-Access     ATA      ST31000520AS     CC32 PQ: 0 ANSI: 5
[    2.248026] ahci 0000:02:00.0: controller reset failed (0x80000001)
[    2.248100] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode
[    2.248102] ahci 0000:02:00.0: flags: 
[    2.248239] scsi6 : ahci
[    2.248291] ata7: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75
[    2.557575] ata7: SATA link down (SStatus 0 SControl 300)
[    2.565869] sd 4:0:0:0: [sde] Attached SCSI disk
[    2.579963] sd 5:0:0:0: [sdf] Attached SCSI disk
[    2.585895] sd 1:0:0:0: [sdb] Attached SCSI disk
[    2.589825] sd 2:0:0:0: [sdc] Attached SCSI disk
[    3.039930] sd 3:0:0:0: [sdd] Attached SCSI disk
[    3.083346] sd 0:0:0:0: [sda] Attached SCSI disk
[    3.863782] scsi7 : pata_marvell
[    3.864187] scsi8 : pata_marvell
[    3.864574] ata8: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[    3.864577] ata9: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  165.303195] ata9: soft resetting link
[  165.465217] ata9: EH complete
[  179.817277] ata8: soft resetting link
[  179.969106] ata8: EH complete
[  189.161840] ata7: hard resetting link
[  189.467441] ata7: SATA link down (SStatus 0 SControl 300)
[  189.467461] ata7: EH complete
[  315.833140] ata8: soft resetting link
[  315.984546] ata8: EH complete
[  318.974146] ata8: soft resetting link
[  319.125082] ata8: EH complete
[  328.019533] scsi9 : pata_marvell
[  328.019951] scsi10 : pata_marvell
[  328.020417] ata10: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[  328.020426] ata11: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  373.887292] ata11: soft resetting link
[  374.048856] ata11: EH complete
[  381.099605] ata10: soft resetting link
[  381.250912] ata10: EH complete
[  409.101457] ahci 0000:02:00.0: irq 75 for MSI/MSI-X
[  409.101485] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1
[  410.106469] ahci 0000:02:00.0: controller reset failed (0x80000003)
[  410.106481] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode
[  410.106483] ahci 0000:02:00.0: flags: 
[  410.106655] scsi11 : ahci
[  410.106706] ata12: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75
[  410.412017] ata12: SATA link down (SStatus 0 SControl 300)
[  427.452877] ata12: hard resetting link
[  427.757703] ata12: SATA link down (SStatus 0 SControl 300)
[  427.757721] ata12: EH complete
[  443.326795] scsi12 : pata_marvell
[  443.327090] scsi13 : pata_marvell
[  443.327348] ata13: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[  443.327354] ata14: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  455.198604] ata13: soft resetting link
[  455.349916] ata13: EH complete
[  460.648310] ata14: soft resetting link
[  460.809941] ata14: EH complete

As you can see, I played around with the different drivers (pata-marvell loads normally now), but no devices are recognized. Windows VM still BSODs.

---

i'm sorry for my poor english wirting skills…

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Yep, that probably explains it.  I'll create fresh VMs (Windows and Ubuntu) using sata/ahci next week and will test whether they too run with the unpatched versions of qemu and seabios along with the stock Ubuntu 3.10 kernel.

Oh, I tested my Nvidia GT610 as yet another secondary GPU, and it seems to be working great too.  Not quite as fast as my Nvidia GTX550Ti, but plenty  fast for Netflix and Youtube.

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor, also dont forget to check out my post on performance report/improvements https://bbs.archlinux.org/viewtopic.php … 1#p1270311

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

So it does work at some point but after doing a reboot it doesnt work anymore?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I seem to be having the same issue that apoapo and myweb are running through.

- With the Radeon as the primary video device, I get apoapo's dirty stream of errors :

system-x86_64: vfio_bar_write(,0x13ff50, 0x36703067,4) failed: Device or resource busy

and this in dmesg:

[  763.408269] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]
[  763.408280] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]
[  763.408291] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

With Intel integrated as the primary device, no errors come up, but the Radeon never initializes.

The latter seems like an easier start point. Is there any way to force that card to turn on during the SEABios boot? Any way to log why it's not doing so?

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

Last edited by nbhs (2013-07-27 19:35:26)

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci

parm:           marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add  ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

I read that wikipedia article a while back, but it doesn't work. With ahci.marvell_enable=1 the driver even gets loaded automatically, but it doesn't recognize any devices (at least with iommu on, but the pata-marvell driver works decent with iommu off, so why try to force the ahci?). Relevant dmesg (dmesg|egrep -i (ata|ahci|scsi))

[    0.955526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.990232] SCSI subsystem initialized
[    0.993158] ACPI: bus type ATA registered
[    0.993369] libata version 3.00 loaded.
[    0.994387] pata_acpi 0000:02:00.1: enabling device (0000 -> 0003)
[    1.238497] ahci 0000:00:11.0: version 3.0
[    1.238729] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.238734] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
[    1.240134] scsi0 : ahci
[    1.240202] scsi1 : ahci
[    1.240277] scsi2 : ahci
[    1.240333] scsi3 : ahci
[    1.240418] scsi4 : ahci
[    1.240470] scsi5 : ahci
[    1.240513] ata1: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19
[    1.240515] ata2: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19
[    1.240517] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19
[    1.240519] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19
[    1.240521] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19
[    1.240523] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19
[    1.240619] ahci 0000:02:00.0: irq 75 for MSI/MSI-X
[    1.240632] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1
[    1.702088] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.702107] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.702123] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.703241] ata6.00: ATA-8: ST31000520AS, CC32, max UDMA/133
[    1.703243] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    1.704090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.704561] ata6.00: configured for UDMA/133
[    1.705090] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.705109] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.705619] ata1.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133
[    1.705620] ata1.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.706205] ata1.00: configured for UDMA/133
[    1.706285] scsi 0:0:0:0: Direct-Access     ATA      WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5
[    1.708208] ata3.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133
[    1.708210] ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.710489] ata2.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7
[    1.710491] ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.711470] ata4.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133
[    1.711472] ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[    1.713723] ata5.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133
[    1.713725] ata5.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    1.714324] ata3.00: configured for UDMA/133
[    1.716961] ata2.00: configured for UDMA/133
[    1.717032] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD103SI  1AG0 PQ: 0 ANSI: 5
[    1.717202] scsi 2:0:0:0: Direct-Access     ATA      SAMSUNG HD204UI  1AQ1 PQ: 0 ANSI: 5
[    1.718470] ata4.00: configured for UDMA/133
[    1.718535] scsi 3:0:0:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[    1.723710] ata5.00: configured for UDMA/133
[    1.723771] scsi 4:0:0:0: Direct-Access     ATA      ADATA SP900      5.0. PQ: 0 ANSI: 5
[    1.723902] scsi 5:0:0:0: Direct-Access     ATA      ST31000520AS     CC32 PQ: 0 ANSI: 5
[    2.248026] ahci 0000:02:00.0: controller reset failed (0x80000001)
[    2.248100] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode
[    2.248102] ahci 0000:02:00.0: flags: 
[    2.248239] scsi6 : ahci
[    2.248291] ata7: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75
[    2.557575] ata7: SATA link down (SStatus 0 SControl 300)
[    2.565869] sd 4:0:0:0: [sde] Attached SCSI disk
[    2.579963] sd 5:0:0:0: [sdf] Attached SCSI disk
[    2.585895] sd 1:0:0:0: [sdb] Attached SCSI disk
[    2.589825] sd 2:0:0:0: [sdc] Attached SCSI disk
[    3.039930] sd 3:0:0:0: [sdd] Attached SCSI disk
[    3.083346] sd 0:0:0:0: [sda] Attached SCSI disk
[    3.863782] scsi7 : pata_marvell
[    3.864187] scsi8 : pata_marvell
[    3.864574] ata8: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[    3.864577] ata9: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  165.303195] ata9: soft resetting link
[  165.465217] ata9: EH complete
[  179.817277] ata8: soft resetting link
[  179.969106] ata8: EH complete
[  189.161840] ata7: hard resetting link
[  189.467441] ata7: SATA link down (SStatus 0 SControl 300)
[  189.467461] ata7: EH complete
[  315.833140] ata8: soft resetting link
[  315.984546] ata8: EH complete
[  318.974146] ata8: soft resetting link
[  319.125082] ata8: EH complete
[  328.019533] scsi9 : pata_marvell
[  328.019951] scsi10 : pata_marvell
[  328.020417] ata10: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[  328.020426] ata11: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  373.887292] ata11: soft resetting link
[  374.048856] ata11: EH complete
[  381.099605] ata10: soft resetting link
[  381.250912] ata10: EH complete
[  409.101457] ahci 0000:02:00.0: irq 75 for MSI/MSI-X
[  409.101485] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1
[  410.106469] ahci 0000:02:00.0: controller reset failed (0x80000003)
[  410.106481] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode
[  410.106483] ahci 0000:02:00.0: flags: 
[  410.106655] scsi11 : ahci
[  410.106706] ata12: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75
[  410.412017] ata12: SATA link down (SStatus 0 SControl 300)
[  427.452877] ata12: hard resetting link
[  427.757703] ata12: SATA link down (SStatus 0 SControl 300)
[  427.757721] ata12: EH complete
[  443.326795] scsi12 : pata_marvell
[  443.327090] scsi13 : pata_marvell
[  443.327348] ata13: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45
[  443.327354] ata14: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45
[  455.198604] ata13: soft resetting link
[  455.349916] ata13: EH complete
[  460.648310] ata14: soft resetting link
[  460.809941] ata14: EH complete

As you can see, I played around with the different drivers (pata-marvell loads normally now), but no devices are recognized. Windows VM still BSODs.

I'm unsure what your goal is meanwhile. If your goal is to get it passed through to the VM, I'll summarize what I did to get it working in Win7 without issues. If your goal is to get it working in Linux while IOMMU is enabled in the BIOS, I can't help.

My goal was to get it passed through to the VM, in AHCI mode of course, as the controller is completely useless in the host anyway due to, let's call it "the missing workaround" in the amd-iommu driver.
I didn't test the controller in IDE mode, because IDE emulation is worthless (I'd rather pass through a disk attached to my internal 5-port JMB393 controller than one in IDE emulation mode on the Marvell SATA3 controller) - I do not even have the pata-marvell driver enabled in my kernel config for that reason. So I

* set the controller to AHCI mode in UEFI/BIOS (actually no UEFI here, the Gigabyte has a legacy BIOS mode)
* added its vendor:id to pci-stub.ids boot parameter (as said, it's completely useless for the host anyways, so make the AHCI driver ignore it completely)
* passed it through to the qemu VM using "-device vfio-pci,host=${SATA_DEVICE},bus=pcie.0" (actually it shows up with two entries in lscpi, one of wich seems to be for the E-SATA ports (they also do not work in the Linux host, same issues))
* Win7 automatically installed some ATA drivers for it (I didn't download anything anywhere)
* it works, without issues

Why is it that you refuse to bind it to pci-stub and instead mess around with blacklisting drivers in different modes? I don't get it, please excuse me if I just missed your point or got something wrong from your posts.

Last edited by teekay (2013-07-27 20:00:40)

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm unsure what your goal is meanwhile. If your goal is to get it passed through to the VM, I'll summarize what I did to get it working in Win7 without issues. If your goal is to get it working in Linux while IOMMU is enabled in the BIOS, I can't help.

My goal was to get it passed through to the VM, in AHCI mode of course, as the controller is completely useless in the host anyway due to, let's call it "the missing workaround" in the amd-iommu driver.
I didn't test the controller in IDE mode, because IDE emulation is worthless (I'd rather pass through a disk attached to my internal 5-port JMB393 controller than one in IDE emulation mode). I do not even have the pata-marvell driver enabled in my kernel config for that reason. To sum it up, the marvell controller is completely useless for the host OS here. So I

* set the controller to AHCI mode in UEFI/BIOS (actually no UEFI here, the Gigabyte has a legacy BIOS mode)
* added its vendor:id to pci-stub.ids boot parameter (as said, it's completely useless for the host anyways, so make the AHCI driver ignore it completely)
* passed it through to the qemu VM using "-device vfio-pci,host=${SATA_DEVICE},bus=pcie.0" (actually it shows up with two entries in lscpi, one of wich seems to be for the E-SATA ports (they also do not work in the Linux host, same issues))
* Win7 automatically installed some ATA drivers for it (I didn't download anything anywhere)
* it works, without issues

Why is it that you refuse to bind it to pci-stub and instead mess around with blacklisting drivers in different modes? I don't get it, please excuse my ignorance.

My goal is to get it passed through to the VM, but IOMMU has to be enabled in order to to that. I play around with the controller on the host, because I can't imagine, that the controller will work in a guest, bot not work on the host, while it works perfectly fine on the host with IOMMU disabled, but refuses to work with IOMMU enabled. So why should it work in the guest?
Also, where's the difference between loading no driver at all and binding to pci-stub? While pata-marvell was blacklisted and I hadn't added ahci.marvell_enable=1 to my kernel parameters no driver got loaded (see lspci -vvvnn earlier…).
I'll try pci-stub later, but we don't have the exact same controller (both are buggy, but maybe in different ways) and boards from a different manufacturer, so different things might work or not.
Also, the Windows installer doesn't even find drives on mine without loading a special driver from the mainboard dvd, was that the same for yours? My intention is to boot with this controller from a SSD, because trim and stuff might be strange through virtio.

Note about the E-SATA ports: I think they are handled by the same controller, arent they?

---

i'm sorry for my poor english wirting skills…

GizmoChicken

Member

Registered: 2013-07-25

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor.

I was afraid of that.  Don't know if my borderline obsessive-compulsive disorder would permit me to use different venders. :-)  That, and as much as I am loathe to compliment Asus, their version probably has among best cooling fins for the price.  I'll look into the possibility of flashing one card with a modified vendor ID.

It's a shame that we can't use pciback, in place of pci-stub, to grab the GPU before a driver binds it.  I found an outdated Xen wiki entry discussing how pciback could be loaded (presumably in a Xen environment) as module if not recompiled in the kernel.

I don't suppose you've heard of anyone getting the pciback module (or a derivative thereof) to load outside the Xen environment?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor.

I was afraid of that.  Don't know if my borderline obsessive-compulsive disorder would permit me to use different venders. :-)  That, and as much as I am loathe to compliment Asus, their version probably has among best cooling fins for the price.  I'll look into the possibility of flashing one card with a modified vendor ID.

It's a shame that we can't use pciback, in place of pci-stub, to grab the GPU before a driver binds it.  I found an outdated Xen wiki entry discussing how pciback could be loaded (presumably in a Xen environment) as module if not recompiled in the kernel.

I don't suppose you've heard of anyone getting the pciback module (or a derivative thereof) to load outside the Xen environment?

Well now that i think about it it might work if you bind it manually to the pci-stub module but to avoid problems just flash the bios

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My goal is to get it passed through to the VM, but IOMMU has to be enabled in order to to that. I play around with the controller on the host, because I can't imagine, that the controller will work in a guest, bot not work on the host, while it works perfectly fine on the host with IOMMU disabled, but refuses to work with IOMMU enabled. So why should it work in the guest?

Well, my only explanation is: it's the driver that fails. My board has some Etron USB3 controller. It works perfectly fine in Linux, but the driver for Windows 7 caused BSOD gallore, no matter if it's in a VM or a "real" install. So that's a counter example for that Marvell controller which, if IOMMU is enabled, totally fails in Linux, but works fine in Windows. You did find the bug report and the post about the intel-iommu driver having a workaround, but not the amd-iommu one. (I found the same posts when I was looking for the problem)

Also, where's the difference between loading no driver at all and binding to pci-stub? While pata-marvell was blacklisted and I hadn't added ahci.marvell_enable=1 to my kernel parameters no driver got loaded (see lspci -vvvnn earlier…).

I have no idea, but my thinking was: if it (currently) doesn't work anyway in the host and just causes page fault and what not, and there is a way to make the host ignore it before any driver even tries to act on it (disabling AHCI driver is no option), I'll use it, Yay for pci-stub.

I'll try pci-stub later, but we don't have the exact same controller (both are buggy, but maybe in different ways) and boards from a different manufacturer, so different things might work or not.

I realized that from your lscpi output, yes. But as they seem to act almost the same in the Linux, it's worth a try for you.

Also, the Windows installer doesn't even find drives on mine without loading a special driver from the mainboard dvd, was that the same for yours? My intention is to boot with this controller from a SSD, because trim and stuff might be strange through virtio.

I haven't tried that. My Windows VM install is actually a converted VirtualBox image, with vbox tools uninstalled, virtio drivers added, and the install itself now resides on a qcow image. The Marvell SATA only provides a data disk inside the VM. After I added it in Qemu, Windows automatically installed drivers for it, and it "just worked"™

Note about the E-SATA ports: I think they are handled by the same controller, arent they?

The same driver, yes (ahci if running in AHCI mode), but on my board the marvell exhibits itself as 2 separate controllers in lspci, and I _believe_ one of them is for the e-sata ports, I haven't verified that though (I replaced my external RAID bay with a larger case and an internal port multiplier, that external IcyDock thing was way too noisy). I pci-stub'ed both (as the have same vendor and id), but maybe I just passed through the right one by pure luck.

Last edited by teekay (2013-07-27 21:34:40)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

pcworld

Member

From: Germany

Registered: 2013-07-28

Posts: 6

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one.

Does that mean VGA passthrough worked for the GTX 650 (as the guest)?

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Not using this method, if you had a radeon card you could use kvm pci-assign and pass it as secondary but on NV cards this is the only reliable method (unless you're willing to try xen, and its counteless unreliable hacks)

Is there any other method that works with only one Nvidia card and KVM? It seems Xen works with only one graphics card, when you boot the VM your graphics card is passed through to the guest system. (If Xen worked properly with Nvidia cards, that is.)
The host system could still be controlled via VNC/X Forwarding or similarly, and when I shut down the guest system I'd like the graphics card to be reassigned to my host system.

I'm considering buying a GTX 650 Ti Boost, since many (fanboys?) say Nvidia's proprietary Linux drivers were much better than fglrx (I don't know if it's true).
On the CPU side, the AMD FX-8350 and Intel i7-2600 are on my list, though I'm rather leaning towards the AMD one since it's cheaper and officially supports overclocking. Advantage of the i7 would be its integrated GPU, but apparently it's not too reliable if it's your primary GPU and you want to forward/VGA-passthrough the other one.

Edit: Of course, great thanks for your tutorial! I've come across many VGA-passthrough threads (mostly concering Xen), and this one seems to be one of very few that are not dead (yet).

Last edited by pcworld (2013-07-28 11:09:37)

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

- I have the SeaBIOS 1.7.2 w/ patches (extension dirty? is that the right one?)
- Is there a patch for Kernel 3.9? I can't update to 3.10 as it breaks DKMS, and ZFS-DKMS runs my primary storage
- Do I need a patch Qemu or is the latest git version already updated for VFIO-reset? If not, which version do I clone to patch?

I apologize for putting my noobishness on front row here, but if anyone has a command set for pulling down and patching the kernel/qemu, I'd much appreciate it.

Last edited by mukiex (2013-07-28 14:22:42)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

Last edited by myweb (2013-07-28 14:38:48)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

Last edited by nbhs (2013-07-28 19:16:06)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

- I have the SeaBIOS 1.7.2 w/ patches (extension dirty? is that the right one?)
- Is there a patch for Kernel 3.9? I can't update to 3.10 as it breaks DKMS, and ZFS-DKMS runs my primary storage
- Do I need a patch Qemu or is the latest git version already updated for VFIO-reset? If not, which version do I clone to patch?

I apologize for putting my noobishness on front row here, but if anyone has a command set for pulling down and patching the kernel/qemu, I'd much appreciate it.

Right now you'll need a patched seabios, patched kernel, and patched qemu until they get merged upstream, using archlinux you can use my builds, you can fetch the kernel and qemu from alex williamson's trees https://github.com/awilliam/qemu-vfio and https://github.com/awilliam/linux-vfio branch vga-current

Last edited by nbhs (2013-07-28 19:13:20)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one.

Does that mean VGA passthrough worked for the GTX 650 (as the guest)?

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Not using this method, if you had a radeon card you could use kvm pci-assign and pass it as secondary but on NV cards this is the only reliable method (unless you're willing to try xen, and its counteless unreliable hacks)

Is there any other method that works with only one Nvidia card and KVM? It seems Xen works with only one graphics card, when you boot the VM your graphics card is passed through to the guest system. (If Xen worked properly with Nvidia cards, that is.)
The host system could still be controlled via VNC/X Forwarding or similarly, and when I shut down the guest system I'd like the graphics card to be reassigned to my host system.

I'm considering buying a GTX 650 Ti Boost, since many (fanboys?) say Nvidia's proprietary Linux drivers were much better than fglrx (I don't know if it's true).
On the CPU side, the AMD FX-8350 and Intel i7-2600 are on my list, though I'm rather leaning towards the AMD one since it's cheaper and officially supports overclocking. Advantage of the i7 would be its integrated GPU, but apparently it's not too reliable if it's your primary GPU and you want to forward/VGA-passthrough the other one.

Edit: Of course, great thanks for your tutorial! I've come across many VGA-passthrough threads (mostly concering Xen), and this one seems to be one of very few that are not dead (yet).

You'll need 2 cards one for the host, and one for the vm, right now amd cpus seems more successful at getting it working

Last edited by nbhs (2013-07-28 19:16:49)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

I see windows installation loading logo on passthrough'd card and then BSOD on passthrough'd card. BSOD: ACPI_BIOS_ERROR
SeaBIOS compiled from git: 1.7.3-18-g7093aa5

Also I see the following message in console when starting QEMU:
Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.

Last edited by myweb (2013-07-28 21:03:44)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

I see windows installation loading logo on passthrough'd card and then BSOD on passthrough'd card. BSOD: ACPI_BIOS_ERROR
SeaBIOS compiled from git: 1.7.3-18-g7093aa5

Also I see the following message in console when starting QEMU:
Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Last edited by nbhs (2013-07-28 21:27:07)

BulliteShot

Member

Registered: 2013-07-28

Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying to get this working for days now.. works on Xen but it's not usable due to all the crashing and poor performance.

I've had enough of my intel board. Looking to buy a AM3+ gigabyte board... anyone have any suggestions that works with this?

EDIT: Also, I would like the integrated AMD graphics to work on the host.

Last edited by BulliteShot (2013-07-28 21:34:23)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying to get this working for days now.. works on Xen but it's not usable due to all the crashing and poor performance.

I've had enough of my intel board. Looking to buy a AM3+ gigabyte board... anyone have any suggestions that works with this?

EDIT: Also, I would like the integrated AMD graphics to work on the host.

Have you tried this guide on your intel build?

BulliteShot

Member

Registered: 2013-07-28

Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.021516] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.297980] IOMMU 0 0xfed90000: using Queued invalidation
[    0.297981] IOMMU 1 0xfed91000: using Queued invalidation
[    0.436381] IOMMU: software identity mapping for device 0000:00:00.0
[    0.436382] IOMMU: software identity mapping for device 0000:00:01.0
[    0.436383] IOMMU: software identity mapping for device 0000:00:14.0
[    0.436384] IOMMU: software identity mapping for device 0000:00:16.0
[    0.436385] IOMMU: software identity mapping for device 0000:00:1a.0
[    0.436386] IOMMU: software identity mapping for device 0000:00:1b.0
[    0.436387] IOMMU: software identity mapping for device 0000:00:1c.0
[    0.436388] IOMMU: software identity mapping for device 0000:00:1c.2
[    0.436389] IOMMU: software identity mapping for device 0000:00:1d.0
[    0.436390] IOMMU: software identity mapping for device 0000:00:1f.0
[    0.436391] IOMMU: software identity mapping for device 0000:00:1f.2
[    0.436392] IOMMU: software identity mapping for device 0000:00:1f.3
[    0.436396] IOMMU: software identity mapping for device 0000:01:00.0
[    0.436397] IOMMU: software identity mapping for device 0000:01:00.1
[    0.436401] IOMMU: software identity mapping for device 0000:02:00.0
[    0.436402] IOMMU: Setting RMRR:
[    0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff]
[    0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff]
[    0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff]
[    0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[root@localhost ~]# 

[root@localhost ~]# dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt
[    0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.705438] pci-stub 0000:01:00.0: claimed by stub
[    0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.705448] pci-stub 0000:01:00.1: claimed by stub
[root@localhost ~]# 

[   60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 2 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device virtio-scsi-pci,id=scsi \
-drive if=none,format=raw,file=/root/win_os.raw,id=disk \
-device scsi-hd,drive=disk \
-net nic,model=virtio \
-net tap,ifname=vm-win,script=no,downscript=no

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.021516] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.297980] IOMMU 0 0xfed90000: using Queued invalidation
[    0.297981] IOMMU 1 0xfed91000: using Queued invalidation
[    0.436381] IOMMU: software identity mapping for device 0000:00:00.0
[    0.436382] IOMMU: software identity mapping for device 0000:00:01.0
[    0.436383] IOMMU: software identity mapping for device 0000:00:14.0
[    0.436384] IOMMU: software identity mapping for device 0000:00:16.0
[    0.436385] IOMMU: software identity mapping for device 0000:00:1a.0
[    0.436386] IOMMU: software identity mapping for device 0000:00:1b.0
[    0.436387] IOMMU: software identity mapping for device 0000:00:1c.0
[    0.436388] IOMMU: software identity mapping for device 0000:00:1c.2
[    0.436389] IOMMU: software identity mapping for device 0000:00:1d.0
[    0.436390] IOMMU: software identity mapping for device 0000:00:1f.0
[    0.436391] IOMMU: software identity mapping for device 0000:00:1f.2
[    0.436392] IOMMU: software identity mapping for device 0000:00:1f.3
[    0.436396] IOMMU: software identity mapping for device 0000:01:00.0
[    0.436397] IOMMU: software identity mapping for device 0000:01:00.1
[    0.436401] IOMMU: software identity mapping for device 0000:02:00.0
[    0.436402] IOMMU: Setting RMRR:
[    0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff]
[    0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff]
[    0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff]
[    0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[root@localhost ~]# 

[root@localhost ~]# dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt
[    0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.705438] pci-stub 0000:01:00.0: claimed by stub
[    0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.705448] pci-stub 0000:01:00.1: claimed by stub
[root@localhost ~]# 

[   60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 2 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device virtio-scsi-pci,id=scsi \
-drive if=none,format=raw,file=/root/win_os.raw,id=disk \
-device scsi-hd,drive=disk \
-net nic,model=virtio \
-net tap,ifname=vm-win,script=no,downscript=no

Are you using the lastest builds i posted?  iommu=pt is an amd thing only (i think for intel is intel_iommu=pt) and i have no idea what igfx_on does, it seems myweb has partially got it to work on his intel board (except the bsod which im sure its a seabios thing), perhaps he knows more since i dont have an intel board to test

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Yes, You are right - I have no BSOD with 1.7.2 but I could not apply kernel-vfio-vga-reset.patch to Linux kernel from vga-current branch: I get

Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej
patching file drivers/pci/hotplug/pciehp_core.c
Reversed (or previously applied) patch detected!  Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej
patching file drivers/pci/hotplug/pciehp_hpc.c
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej
patching file drivers/pci/pci.c
Reversed (or previously applied) patch detected!  Skipping patch.
5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej
patching file drivers/vfio/pci/vfio_pci.c
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej
patching file drivers/vfio/vfio.c
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej
patching file include/linux/pci.h
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej
patching file include/linux/pci_hotplug.h
Reversed (or previously applied) patch detected!  Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej
patching file include/linux/vfio.h
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej
patching file include/uapi/linux/vfio.h
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej
==> ERROR: A failure occurred in prepare().
    Aborting...

Last edited by myweb (2013-07-28 21:58:34)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

So it does work at some point but after doing a reboot it doesnt work anymore?

Yes, You are right but it does not work anymore until VM is not restarted (qemu).
But when I restart Qemu and boot to windows, windows says that installation was unexpectedly interrupted and asks to reinstall.