You are not logged in.

#401 2013-08-02 12:18:16

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

Offline

#402 2013-08-02 12:26:05

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:
cmdr wrote:
nbhs wrote:

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

http://pci-ids.ucw.cz/read/PC/8086/1e1e

Offline

#403 2013-08-02 12:33:07

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:
cmdr wrote:
nbhs wrote:

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

Yes, it's a pci root port/bridge. Before i used 'pcie_acs_override" every device behind that root port/bridge was in the same iommu_group.

# ls -l /sys/kernel/iommu_groups/6/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

With the changed cmdline the devices are now in their own groups:

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 14:32 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 14:32 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 14:32 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

]# ls -l /sys/bus/pci/devices/0000\:05\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 14:32 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0

# ls -l /sys/bus/pci/devices/0000\:0a\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 14:32 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0

Offline

#404 2013-08-02 14:14:22

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I keep forgetting to post this:

If your Intel system won't boot with iommu=on enabled, it's a bug. Straight-up. I've gotten Xen working with passthrough PCI (though not at DomU boot) as far back as Ubuntu 12.04. However, under KVM at the moment, setting that option will result in your system not booting if you have a 0000:05:00.0 pci device. I tried other options, but that's basically what it came down to. Remove the extra PCI device. It will work. It's messed up, I know.

Offline

#405 2013-08-02 14:32:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

I keep forgetting to post this:

If your Intel system won't boot with iommu=on enabled, it's a bug. Straight-up. I've gotten Xen working with passthrough PCI (though not at DomU boot) as far back as Ubuntu 12.04. However, under KVM at the moment, setting that option will result in your system not booting if you have a 0000:05:00.0 pci device. I tried other options, but that's basically what it came down to. Remove the extra PCI device. It will work. It's messed up, I know.

It's not a bug, to enable the iommu on intel the correct kernel parameter is intel_iommu=on, iommu=on might be a xen parameter but its completely unrelated to kvm, since xen is a type1 hypervisor with its own kernel.

Offline

#406 2013-08-02 15:13:59

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

I installed the ACS-patch also and now everything works! Also the 4x PCI slots now have the devices in their own iommu groups. Thanks for this info. So now I have 2 VM's both with their own graphics cards on top of the host smile. Arch + Windows 8 + Ubuntu 12.04.

Now I need another card to test 4 ... (and a PCIe extender cable, since I've run out of room a bit in the card department..)

Last edited by reintseri (2013-08-02 15:14:44)

Offline

#407 2013-08-02 22:00:46

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

Note that I've tried pcie_acs_override = downstream, multifunciton, and singling out the vendor/device numbers for one of the root port. This output did not change.
iommu grouping on my motherboard is insane. If the videocard is on the first PCI express port, I only get the video card and the root PCI-E port. However, moving the card to the other port gives me this.

Via  /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ :

lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.4 -> ../../../../devices/pci0000:00/0000:00:1c.4
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.6 -> ../../../../devices/pci0000:00/0000:00:1c.6
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0

lspci for those devices:

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
03:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)

Offline

#408 2013-08-02 22:29:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey guys,

Note that I've tried pcie_acs_override = downstream, multifunciton, and singling out the vendor/device numbers for one of the root port. This output did not change.
iommu grouping on my motherboard is insane. If the videocard is on the first PCI express port, I only get the video card and the root PCI-E port. However, moving the card to the other port gives me this.

Via  /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ :

lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.4 -> ../../../../devices/pci0000:00/0000:00:1c.4
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:00:1c.6 -> ../../../../devices/pci0000:00/0000:00:1c.6
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  2 16:52 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0

lspci for those devices:

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
03:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)

Sorry to ask but did you apply the patch first?

Offline

#409 2013-08-02 23:08:09

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

Offline

#410 2013-08-02 23:10:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

No its not, its here: http://www.spinics.net/lists/kvm/msg92163.html or here: http://marc.info/?l=linux-kernel&m=136993923507507&w=2 i did not include it since its not in alex tree and technically its unrelated

Last edited by nbhs (2013-08-02 23:12:11)

Offline

#411 2013-08-03 19:47:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
mukiex wrote:

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

No its not, its here: http://www.spinics.net/lists/kvm/msg92163.html or here: http://marc.info/?l=linux-kernel&m=136993923507507&w=2 i did not include it since its not in alex tree and technically its unrelated

I re uploaded the linux-mainline package to include this patch its on the first page

Offline

#412 2013-08-04 11:29:05

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I re uploaded the linux-mainline package to include this patch its on the first page

Thank you so much! Adding "pcie_acs_override=downstream" took care of the iommu_groups silliness. Now only the graphics card pieces show up.

The blue screen seems to happen across multiple driver versions (tried about 4 different drivers between the month my card launched and the latest beta). Same file: atikmpag.sys

Now, I don't know if this helps, but the situation is also bad with an Nvidia card: I swapped it out, reverted to a fresh, driver-less install (yay, ZFS), and installed it. No blue-screen, but also no working driver; driver fails to start and the system reverts to a pre-driver setup.


nbhs: I want to make 110% sure I'm not screwing it up on my end. Each piece of the puzzle I'm running the vfio-reset stuff (built from your packages; thanks again), but I just want to make sure I've installed each bit correctly; is there anything to look out for to verify this?

Last edited by mukiex (2013-08-04 11:32:04)

Offline

#413 2013-08-04 17:23:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
nbhs wrote:

I re uploaded the linux-mainline package to include this patch its on the first page

Thank you so much! Adding "pcie_acs_override=downstream" took care of the iommu_groups silliness. Now only the graphics card pieces show up.

The blue screen seems to happen across multiple driver versions (tried about 4 different drivers between the month my card launched and the latest beta). Same file: atikmpag.sys

Now, I don't know if this helps, but the situation is also bad with an Nvidia card: I swapped it out, reverted to a fresh, driver-less install (yay, ZFS), and installed it. No blue-screen, but also no working driver; driver fails to start and the system reverts to a pre-driver setup.


nbhs: I want to make 110% sure I'm not screwing it up on my end. Each piece of the puzzle I'm running the vfio-reset stuff (built from your packages; thanks again), but I just want to make sure I've installed each bit correctly; is there anything to look out for to verify this?

Have you tried with windows 8?

Offline

#414 2013-08-04 23:14:31

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried Windows 7 64Bit and the vm runs fine. But when i install the latest ATI driver (ONLY the display driver) i get a bsod on reboot. Downloading Windows 8 now; 6 hours left...

Offline

#415 2013-08-06 13:33:46

licious
Member
Registered: 2013-08-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Specs :
cpu:Intel i7 3770
motherboard : Asus P8H77-I
host gpu : Intel HD 4000
guest gpu : Nvidia GT 620
Host : Ubuntu 13.04 with patched kernel 3.10.1
grub: GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=8086:0151,10de:0f01,10de:0bea vfio_iommu_type1.allow_unsafe_interrupts=1"
guest : Ubuntu 13.04, kernel 3.8.0-27-generic (stock)


I've built the patched kernel, seabios and qemu - but I can't get an output on the screen connected to the GT620 when I use vfio. Qemu output is always blank / black as well. If I use "-device pci-assign" instead of "-device vfio-pci" then my guest os initializes the GT620 when starting X on the guest (no output at all before X starts). I can only get an output if I'm using the Nouveau-drivers on the guest, Nvidia binary drivers won't work. I've tried patching the drivers as well, but I've had no luck in getting the patched drivers to work. The output from dmesg on the guest :
[   17.260770] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[   17.260778] NVRM: rm_init_adapter(0) failed

the weird thing is that it seems like hdmi sound is initialized :
[   13.078985] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input5
[   13.079086] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input6
[   13.079149] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input7
[   13.079225] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input8

Also, when using -device vfio-pci I get an error message saying "qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/xbmc/GT620.rom: Warning, device 0000:01:00.0 does not support reset". This message does not appear when I use pci-assign, most likely because no kernel modules have been loaded for the GPU yet. I thought the patched kernel and/ or qemu would take care of resetting the device as well ?

Offline

#416 2013-08-06 17:58:36

Diggo
Member
Registered: 2013-05-29
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, today i took a chance to migrate to vfio-pci. But i got stuck at the point, where to start the DomU the first time. If i read the posts here right, the patched versions are for VGA-reset only (so for testing, i actually don´t need them?).

The last thing i did was to add the ivrs entry to the commandline. The AMD-Vi error has gone, but still the outcome is the same "Could not initialize SDL(No available video device) - exiting". There must be something I am missing (?) smile

Bash command:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1
Could not initialize SDL(No available video device) - exiting


Some other Outputs:


cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88

lspci:

...
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
...

lspci-n:

01:00.0 0300: 1002:6739
01:00.1 0403: 1002:aa88

Dmesg:

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.5-1-ARCH (tobias@T-POWA-LX) (gcc version 4.8.1 20130725 (prerelease) (GCC) ) #1 SMP PREEMPT Mon Aug 5 08:04:22 CEST 2013
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bda5efff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bda5f000-0x00000000bdaacfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdaad000-0x00000000bdab6fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bdab7000-0x00000000bdab7fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdab8000-0x00000000bdadcfff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdadd000-0x00000000bdaedfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdaee000-0x00000000bdb02fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdb03000-0x00000000bdb04fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdb05000-0x00000000bdb0dfff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdb0e000-0x00000000bdb13fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdb14000-0x00000000bdb75fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdb76000-0x00000000bdd78fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdd79000-0x00000000bdefffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed61000-0x00000000fed70fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100001000-0x000000013fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: To be filled by O.E.M. To be filled by O.E.M./M5A99X EVO, BIOS 1604 10/16/2012
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF write-through
[    0.000000]   C0000-C7FFF write-protect
[    0.000000]   C8000-EBFFF uncachable
[    0.000000]   EC000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
[    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
[    0.000000]   2 base 0000BDF00000 mask FFFFFFF00000 uncachable
[    0.000000]   3 base 0000BE000000 mask FFFFFE000000 uncachable
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] TOM2: 0000000140000000 aka 5120M
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820: update [mem 0xbdf00000-0xffffffff] usable ==> reserved
[    0.000000] e820: last_pfn = 0xbdf00 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x01b22000, 0x01b22fff] PGTABLE
[    0.000000] BRK [0x01b23000, 0x01b23fff] PGTABLE
[    0.000000] BRK [0x01b24000, 0x01b24fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x13fe00000-0x13fffffff]
[    0.000000]  [mem 0x13fe00000-0x13fffffff] page 2M
[    0.000000] BRK [0x01b25000, 0x01b25fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x13c000000-0x13fdfffff]
[    0.000000]  [mem 0x13c000000-0x13fdfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x100001000-0x13bffffff]
[    0.000000]  [mem 0x100001000-0x1001fffff] page 4k
[    0.000000]  [mem 0x100200000-0x13bffffff] page 2M
[    0.000000] BRK [0x01b26000, 0x01b26fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x00100000-0xbda5efff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x3fffffff] page 2M
[    0.000000]  [mem 0x40000000-0x7fffffff] page 1G
[    0.000000]  [mem 0x80000000-0xbd9fffff] page 2M
[    0.000000]  [mem 0xbda00000-0xbda5efff] page 4k
[    0.000000] init_memory_mapping: [mem 0xbdd79000-0xbdefffff]
[    0.000000]  [mem 0xbdd79000-0xbdefffff] page 4k
[    0.000000] RAMDISK: [mem 0x37a58000-0x37d23fff]
[    0.000000] ACPI: RSDP 00000000000f0450 00024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 00000000bdaad070 00054 (v01 ALASKA    A M I 01072009 AMI  00010013)
[    0.000000] ACPI: FACP 00000000bdab5d18 000F4 (v04 ALASKA    A M I 01072009 AMI  00010013)
[    0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130328/tbfadt-603)
[    0.000000] ACPI: DSDT 00000000bdaad158 08BBC (v02 ALASKA    A M I 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000bdb0ef80 00040
[    0.000000] ACPI: APIC 00000000bdab5e10 0009E (v03 ALASKA    A M I 01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 00000000bdab5eb0 0003C (v01 ALASKA    A M I 01072009 MSFT 00010013)
[    0.000000] ACPI: HPET 00000000bdab5ef0 00038 (v01 ALASKA    A M I 01072009 AMI  00000004)
[    0.000000] ACPI: SSDT 00000000bdab6048 008BC (v01 AMD    POWERNOW 00000001 AMD  00000001)
[    0.000000] ACPI: IVRS 00000000bdab5f80 000C8 (v01  AMD     RD890S 00202031 AMD  00000000)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] Scanning NUMA topology in Northbridge 24
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x13fffffff]
[    0.000000]   NODE_DATA [mem 0x13fff7000-0x13fffbfff]
[    0.000000]  [ffffea0000000000-ffffea0004ffffff] PMD -> [ffff88013b600000-ffff88013f5fffff] on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x13fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009cfff]
[    0.000000]   node   0: [mem 0x00100000-0xbda5efff]
[    0.000000]   node   0: [mem 0xbdd79000-0xbdefffff]
[    0.000000]   node   0: [mem 0x100001000-0x13fffffff]
[    0.000000] On node 0 totalpages: 1039233
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3996 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 12080 pages used for memmap
[    0.000000]   DMA32 zone: 773094 pages, LIFO batch:31
[    0.000000]   Normal zone: 4096 pages used for memmap
[    0.000000]   Normal zone: 262143 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 5, version 33, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: IOAPIC (id[0x06] address[0xfec20000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 6, version 33, address 0xfec20000, GSI 24-55
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x43538210 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 4 hotplug CPUs
[    0.000000] nr_irqs_gsi: 72
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000bda5f000 - 00000000bdaad000
[    0.000000] PM: Registered nosave memory: 00000000bdaad000 - 00000000bdab7000
[    0.000000] PM: Registered nosave memory: 00000000bdab7000 - 00000000bdab8000
[    0.000000] PM: Registered nosave memory: 00000000bdab8000 - 00000000bdadd000
[    0.000000] PM: Registered nosave memory: 00000000bdadd000 - 00000000bdaee000
[    0.000000] PM: Registered nosave memory: 00000000bdaee000 - 00000000bdb03000
[    0.000000] PM: Registered nosave memory: 00000000bdb03000 - 00000000bdb05000
[    0.000000] PM: Registered nosave memory: 00000000bdb05000 - 00000000bdb0e000
[    0.000000] PM: Registered nosave memory: 00000000bdb0e000 - 00000000bdb14000
[    0.000000] PM: Registered nosave memory: 00000000bdb14000 - 00000000bdb76000
[    0.000000] PM: Registered nosave memory: 00000000bdb76000 - 00000000bdd79000
[    0.000000] PM: Registered nosave memory: 00000000bdf00000 - 00000000f8000000
[    0.000000] PM: Registered nosave memory: 00000000f8000000 - 00000000fc000000
[    0.000000] PM: Registered nosave memory: 00000000fc000000 - 00000000fec00000
[    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
[    0.000000] PM: Registered nosave memory: 00000000fec01000 - 00000000fec10000
[    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fec11000
[    0.000000] PM: Registered nosave memory: 00000000fec11000 - 00000000fec20000
[    0.000000] PM: Registered nosave memory: 00000000fec20000 - 00000000fec21000
[    0.000000] PM: Registered nosave memory: 00000000fec21000 - 00000000fed00000
[    0.000000] PM: Registered nosave memory: 00000000fed00000 - 00000000fed01000
[    0.000000] PM: Registered nosave memory: 00000000fed01000 - 00000000fed61000
[    0.000000] PM: Registered nosave memory: 00000000fed61000 - 00000000fed71000
[    0.000000] PM: Registered nosave memory: 00000000fed71000 - 00000000fed80000
[    0.000000] PM: Registered nosave memory: 00000000fed80000 - 00000000fed90000
[    0.000000] PM: Registered nosave memory: 00000000fed90000 - 00000000fef00000
[    0.000000] PM: Registered nosave memory: 00000000fef00000 - 0000000100000000
[    0.000000] PM: Registered nosave memory: 0000000100000000 - 0000000100001000
[    0.000000] e820: [mem 0xbdf00000-0xf7ffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] PERCPU: Embedded 29 pages/cpu @ffff88013fc00000 s86144 r8192 d24448 u262144
[    0.000000] pcpu-alloc: s86144 r8192 d24448 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1022972
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ f8000000 size 64 MB
[    0.000000] Memory: 4009760k/5242880k available (4921k kernel code, 1085948k absent, 147172k reserved, 4045k data, 1124k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  Dump stacks of tasks blocking RCU-preempt GP.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8.
[    0.000000] NR_IRQS:4352 nr_irqs:1288 16
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] allocated 16777216 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.003333] tsc: Detected 3009.740 MHz processor
[    0.000003] Calibrating delay loop (skipped), value calculated using timer frequency.. 6021.29 BogoMIPS (lpj=10032466)
[    0.000006] pid_max: default: 32768 minimum: 301
[    0.000033] Security Framework initialized
[    0.000040] AppArmor: AppArmor disabled by boot time parameter
[    0.000265] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.001579] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.002179] Mount-cache hash table entries: 256
[    0.002377] Initializing cgroup subsys memory
[    0.002389] Initializing cgroup subsys devices
[    0.002391] Initializing cgroup subsys freezer
[    0.002392] Initializing cgroup subsys net_cls
[    0.002393] Initializing cgroup subsys blkio
[    0.002416] tseg: 00bdf00000
[    0.002418] CPU: Physical Processor ID: 0
[    0.002419] CPU: Processor Core ID: 0
[    0.002421] mce: CPU supports 6 MCE banks
[    0.002425] LVT offset 0 assigned for vector 0xf9
[    0.002430] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64
tlb_flushall_shift: 4
[    0.002506] Freeing SMP alternatives: 20k freed
[    0.003085] ACPI: Core revision 20130328
[    0.005536] ACPI: All ACPI Tables successfully acquired
[    0.089671] ftrace: allocating 19566 entries in 77 pages
[    0.179081] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.212064] smpboot: CPU0: AMD Athlon(tm) II X4 640 Processor (fam: 10, model: 05, stepping: 03)
[    0.316828] Performance Events: AMD PMU driver.
[    0.316832] ... version:                0
[    0.316833] ... bit width:              48
[    0.316833] ... generic registers:      4
[    0.316834] ... value mask:             0000ffffffffffff
[    0.316835] ... max period:             00007fffffffffff
[    0.316836] ... fixed-purpose events:   0
[    0.316837] ... event mask:             000000000000000f
[    0.336903] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.343565] smpboot: Booting Node   0, Processors  #1 #2 #3
[    0.386697] Brought up 4 CPUs
[    0.386701] smpboot: Total of 4 processors activated (24087.16 BogoMIPS)
[    0.387576] devtmpfs: initialized
[    0.388975] PM: Registering ACPI NVS region [mem 0xbda5f000-0xbdaacfff] (319488 bytes)
[    0.388984] PM: Registering ACPI NVS region [mem 0xbdab7000-0xbdab7fff] (4096 bytes)
[    0.388985] PM: Registering ACPI NVS region [mem 0xbdadd000-0xbdaedfff] (69632 bytes)
[    0.388987] PM: Registering ACPI NVS region [mem 0xbdb03000-0xbdb04fff] (8192 bytes)
[    0.388988] PM: Registering ACPI NVS region [mem 0xbdb0e000-0xbdb13fff] (24576 bytes)
[    0.388990] PM: Registering ACPI NVS region [mem 0xbdb76000-0xbdd78fff] (2109440 bytes)
[    0.389867] RTC time: 20:47:39, date: 08/06/13
[    0.389916] NET: Registered protocol family 16
[    0.390044] node 0 link 0: io port [b000, ffff]
[    0.390046] TOM: 00000000c0000000 aka 3072M
[    0.390048] Fam 10h mmconf [mem 0xe0000000-0xefffffff]
[    0.390050] node 0 link 0: mmio [c0000000, fef0ffff] ==> [c0000000, dfffffff] and [f0000000, fef0ffff]
[    0.390053] TOM2: 0000000140000000 aka 5120M
[    0.390055] bus: [bus 00-1f] on node 0 link 0
[    0.390056] bus: 00 [io  0x0000-0xffff]
[    0.390057] bus: 00 [mem 0xc0000000-0xdfffffff]
[    0.390058] bus: 00 [mem 0xf0000000-0xffffffff]
[    0.390059] bus: 00 [mem 0x140000000-0xfcffffffff]
[    0.390097] ACPI: bus type PCI registered
[    0.390165] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.390167] PCI: not using MMCONFIG
[    0.390168] PCI: Using configuration type 1 for base access
[    0.390169] PCI: Using configuration type 1 for extended access
[    0.390333] mtrr: your CPUs had inconsistent variable MTRR settings
[    0.390334] mtrr: probably your BIOS does not setup all CPUs.
[    0.390335] mtrr: corrected configuration.
[    0.390842] bio: create slab <bio-0> at 0
[    0.391015] ACPI: Added _OSI(Module Device)
[    0.391016] ACPI: Added _OSI(Processor Device)
[    0.391018] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.391019] ACPI: Added _OSI(Processor Aggregator Device)
[    0.391718] ACPI: EC: Look up EC in DSDT
[    0.392488] ACPI: Executed 2 blocks of module-level executable AML code
[    0.407260] ACPI: Interpreter enabled
[    0.407272] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130328/hwxface-568)
[    0.407283] ACPI: (supports S0 S1 S3 S4 S5)
[    0.407285] ACPI: Using IOAPIC for interrupt routing
[    0.407393] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.407424] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources
[    0.419889] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.419975] ACPI: No dock devices found.
[    0.432894] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.433116] PCI host bridge to bus 0000:00
[    0.433120] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.433122] pci_bus 0000:00: root bus resource [io  0x0000-0x03af]
[    0.433123] pci_bus 0000:00: root bus resource [io  0x03e0-0x0cf7]
[    0.433125] pci_bus 0000:00: root bus resource [io  0x03b0-0x03df]
[    0.433126] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.433128] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    0.433130] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff]
[    0.433131] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xffffffff]
[    0.433142] pci 0000:00:00.0: [1002:5a14] type 00 class 0x060000
[    0.433227] pci 0000:00:00.2: [1002:5a23] type 00 class 0x080600
[    0.433331] pci 0000:00:02.0: [1002:5a16] type 01 class 0x060400
[    0.433360] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    0.433392] pci 0000:00:02.0: System wakeup disabled by ACPI
[    0.433421] pci 0000:00:04.0: [1002:5a18] type 01 class 0x060400
[    0.433449] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
[    0.433480] pci 0000:00:04.0: System wakeup disabled by ACPI
[    0.433507] pci 0000:00:06.0: [1002:5a1a] type 01 class 0x060400
[    0.433534] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
[    0.433567] pci 0000:00:06.0: System wakeup disabled by ACPI
[    0.433595] pci 0000:00:0a.0: [1002:5a1d] type 01 class 0x060400
[    0.433622] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[    0.433659] pci 0000:00:0a.0: System wakeup disabled by ACPI
[    0.433693] pci 0000:00:11.0: [1002:4391] type 00 class 0x010601
[    0.433708] pci 0000:00:11.0: reg 10: [io  0xf040-0xf047]
[    0.433715] pci 0000:00:11.0: reg 14: [io  0xf030-0xf033]
[    0.433722] pci 0000:00:11.0: reg 18: [io  0xf020-0xf027]
[    0.433728] pci 0000:00:11.0: reg 1c: [io  0xf010-0xf013]
[    0.433735] pci 0000:00:11.0: reg 20: [io  0xf000-0xf00f]
[    0.433742] pci 0000:00:11.0: reg 24: [mem 0xfeb0b000-0xfeb0b3ff]
[    0.433816] pci 0000:00:12.0: [1002:4397] type 00 class 0x0c0310
[    0.433826] pci 0000:00:12.0: reg 10: [mem 0xfeb0a000-0xfeb0afff]
[    0.433878] pci 0000:00:12.0: System wakeup disabled by ACPI
[    0.433910] pci 0000:00:12.2: [1002:4396] type 00 class 0x0c0320
[    0.433923] pci 0000:00:12.2: reg 10: [mem 0xfeb09000-0xfeb090ff]
[    0.433982] pci 0000:00:12.2: supports D1 D2
[    0.433983] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    0.434003] pci 0000:00:12.2: System wakeup disabled by ACPI
[    0.434033] pci 0000:00:13.0: [1002:4397] type 00 class 0x0c0310
[    0.434043] pci 0000:00:13.0: reg 10: [mem 0xfeb08000-0xfeb08fff]
[    0.434096] pci 0000:00:13.0: System wakeup disabled by ACPI
[    0.434142] pci 0000:00:13.2: [1002:4396] type 00 class 0x0c0320
[    0.434155] pci 0000:00:13.2: reg 10: [mem 0xfeb07000-0xfeb070ff]
[    0.434214] pci 0000:00:13.2: supports D1 D2
[    0.434215] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    0.434236] pci 0000:00:13.2: System wakeup disabled by ACPI
[    0.434267] pci 0000:00:14.0: [1002:4385] type 00 class 0x0c0500
[    0.434342] pci 0000:00:14.2: [1002:4383] type 00 class 0x040300
[    0.434358] pci 0000:00:14.2: reg 10: [mem 0xfeb00000-0xfeb03fff 64bit]
[    0.434405] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[    0.434419] pci 0000:00:14.2: System wakeup disabled by ACPI
[    0.434445] pci 0000:00:14.3: [1002:439d] type 00 class 0x060100
[    0.434523] pci 0000:00:14.4: [1002:4384] type 01 class 0x060401
[    0.434558] pci 0000:00:14.4: System wakeup disabled by ACPI
[    0.434583] pci 0000:00:14.5: [1002:4399] type 00 class 0x0c0310
[    0.434593] pci 0000:00:14.5: reg 10: [mem 0xfeb06000-0xfeb06fff]
[    0.434652] pci 0000:00:14.5: System wakeup disabled by ACPI
[    0.434680] pci 0000:00:16.0: [1002:4397] type 00 class 0x0c0310
[    0.434690] pci 0000:00:16.0: reg 10: [mem 0xfeb05000-0xfeb05fff]
[    0.434746] pci 0000:00:16.0: System wakeup disabled by ACPI
[    0.434777] pci 0000:00:16.2: [1002:4396] type 00 class 0x0c0320
[    0.434791] pci 0000:00:16.2: reg 10: [mem 0xfeb04000-0xfeb040ff]
[    0.434850] pci 0000:00:16.2: supports D1 D2
[    0.434851] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[    0.434874] pci 0000:00:16.2: System wakeup disabled by ACPI
[    0.434903] pci 0000:00:18.0: [1022:1200] type 00 class 0x060000
[    0.434964] pci 0000:00:18.1: [1022:1201] type 00 class 0x060000
[    0.435018] pci 0000:00:18.2: [1022:1202] type 00 class 0x060000
[    0.435071] pci 0000:00:18.3: [1022:1203] type 00 class 0x060000
[    0.435127] pci 0000:00:18.4: [1022:1204] type 00 class 0x060000
[    0.435223] pci 0000:01:00.0: [1002:6739] type 00 class 0x030000
[    0.435237] pci 0000:01:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.435249] pci 0000:01:00.0: reg 18: [mem 0xfea20000-0xfea3ffff 64bit]
[    0.435256] pci 0000:01:00.0: reg 20: [io  0xe000-0xe0ff]
[    0.435270] pci 0000:01:00.0: reg 30: [mem 0xfea00000-0xfea1ffff pref]
[    0.435305] pci 0000:01:00.0: supports D1 D2
[    0.435370] pci 0000:01:00.1: [1002:aa88] type 00 class 0x040300
[    0.435385] pci 0000:01:00.1: reg 10: [mem 0xfea40000-0xfea43fff 64bit]
[    0.435450] pci 0000:01:00.1: supports D1 D2
[    0.443351] pci 0000:00:02.0: PCI bridge to [bus 01]
[    0.443358] pci 0000:00:02.0:   bridge window [io  0xe000-0xefff]
[    0.443361] pci 0000:00:02.0:   bridge window [mem 0xfea00000-0xfeafffff]
[    0.443364] pci 0000:00:02.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.443432] pci 0000:02:00.0: [10ec:8168] type 00 class 0x020000
[    0.443445] pci 0000:02:00.0: reg 10: [io  0xd000-0xd0ff]
[    0.443464] pci 0000:02:00.0: reg 18: [mem 0xd0804000-0xd0804fff 64bit pref]
[    0.443476] pci 0000:02:00.0: reg 20: [mem 0xd0800000-0xd0803fff 64bit pref]
[    0.443528] pci 0000:02:00.0: supports D1 D2
[    0.443529] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.453345] pci 0000:00:04.0: PCI bridge to [bus 02]
[    0.453352] pci 0000:00:04.0:   bridge window [io  0xd000-0xdfff]
[    0.453356] pci 0000:00:04.0:   bridge window [mem 0xd0800000-0xd08fffff 64bit pref]
[    0.453436] pci 0000:03:00.0: [197b:2362] type 00 class 0x010185
[    0.453459] pci 0000:03:00.0: reg 10: [io  0xc040-0xc047]
[    0.453470] pci 0000:03:00.0: reg 14: [io  0xc030-0xc033]
[    0.453480] pci 0000:03:00.0: reg 18: [io  0xc020-0xc027]
[    0.453490] pci 0000:03:00.0: reg 1c: [io  0xc010-0xc013]
[    0.453500] pci 0000:03:00.0: reg 20: [io  0xc000-0xc00f]
[    0.453510] pci 0000:03:00.0: reg 24: [mem 0xfe910000-0xfe9101ff]
[    0.453520] pci 0000:03:00.0: reg 30: [mem 0x00000000-0x0000ffff pref]
[    0.453560] pci 0000:03:00.0: PME# supported from D3hot
[    0.463340] pci 0000:00:06.0: PCI bridge to [bus 03]
[    0.463346] pci 0000:00:06.0:   bridge window [io  0xc000-0xcfff]
[    0.463349] pci 0000:00:06.0:   bridge window [mem 0xfe900000-0xfe9fffff]
[    0.463435] pci 0000:04:00.0: [197b:2362] type 00 class 0x010185
[    0.463458] pci 0000:04:00.0: reg 10: [io  0xb040-0xb047]
[    0.463468] pci 0000:04:00.0: reg 14: [io  0xb030-0xb033]
[    0.463478] pci 0000:04:00.0: reg 18: [io  0xb020-0xb027]
[    0.463488] pci 0000:04:00.0: reg 1c: [io  0xb010-0xb013]
[    0.463498] pci 0000:04:00.0: reg 20: [io  0xb000-0xb00f]
[    0.463508] pci 0000:04:00.0: reg 24: [mem 0xfe810000-0xfe8101ff]
[    0.463518] pci 0000:04:00.0: reg 30: [mem 0x00000000-0x0000ffff pref]
[    0.463558] pci 0000:04:00.0: PME# supported from D3hot
[    0.473334] pci 0000:00:0a.0: PCI bridge to [bus 04]
[    0.473340] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    0.473343] pci 0000:00:0a.0:   bridge window [mem 0xfe800000-0xfe8fffff]
[    0.473425] pci 0000:05:05.0: [102b:0519] type 00 class 0x030000
[    0.473439] pci 0000:05:05.0: reg 10: [mem 0xfe710000-0xfe713fff]
[    0.473449] pci 0000:05:05.0: reg 14: [mem 0xd0000000-0xd07fffff pref]
[    0.473490] pci 0000:05:05.0: reg 30: [mem 0xfe700000-0xfe70ffff pref]
[    0.473554] pci 0000:00:14.4: PCI bridge to [bus 05] (subtractive decode)
[    0.473559] pci 0000:00:14.4:   bridge window [mem 0xfe700000-0xfe7fffff]
[    0.473562] pci 0000:00:14.4:   bridge window [mem 0xd0000000-0xd07fffff pref]
[    0.473564] pci 0000:00:14.4:   bridge window [io  0x0000-0x03af] (subtractive decode)
[    0.473565] pci 0000:00:14.4:   bridge window [io  0x03e0-0x0cf7] (subtractive decode)
[    0.473567] pci 0000:00:14.4:   bridge window [io  0x03b0-0x03df] (subtractive decode)
[    0.473568] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff] (subtractive decode)
[    0.473570] pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
[    0.473572] pci 0000:00:14.4:   bridge window [mem 0x000c0000-0x000dffff] (subtractive decode)
[    0.473573] pci 0000:00:14.4:   bridge window [mem 0xc0000000-0xffffffff] (subtractive decode)
[    0.473589] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM
[    0.473591] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08)
[    0.473916] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 7 10 11 14 15) *0
[    0.473968] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 11 14 15) *0
[    0.474019] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 7 10 11 14 15) *0
[    0.474068] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 10 11 14 15) *0
[    0.474108] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 11 14 15) *0
[    0.474139] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 11 14 15) *0
[    0.474171] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 7 10 11 14 15) *0
[    0.474202] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 10 11 14 15) *0
[    0.474297] acpi root: \_SB_.PCI0 notify handler is installed
[    0.474332] Found 1 acpi root devices
[    0.474350] ACPI: EC: GPE = 0xa, I/O: command/status = 0x66, data = 0x62
[    0.474538] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.474543] vgaarb: device added: PCI:0000:05:05.0,decodes=io+mem,owns=io+mem,locks=none
[    0.474544] vgaarb: loaded
[    0.474545] vgaarb: bridge control possible 0000:05:05.0
[    0.474545] vgaarb: bridge control possible 0000:01:00.0
[    0.474583] PCI: Using ACPI for IRQ routing
[    0.480744] PCI: pci_cache_line_size set to 64 bytes
[    0.480829] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff]
[    0.480831] e820: reserve RAM buffer [mem 0xbda5f000-0xbfffffff]
[    0.480832] e820: reserve RAM buffer [mem 0xbdf00000-0xbfffffff]
[    0.480908] NetLabel: Initializing
[    0.480909] NetLabel:  domain hash size = 128
[    0.480909] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.480919] NetLabel:  unlabeled traffic allowed by default
[    0.480939] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.480942] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    0.482962] Switching to clocksource hpet
[    0.486666] pnp: PnP ACPI init
[    0.486681] ACPI: bus type PNP registered
[    0.486790] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved
[    0.486794] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.487224] system 00:01: [io  0x040b] has been reserved
[    0.487226] system 00:01: [io  0x04d6] has been reserved
[    0.487228] system 00:01: [io  0x0c00-0x0c01] has been reserved
[    0.487229] system 00:01: [io  0x0c14] has been reserved
[    0.487231] system 00:01: [io  0x0c50-0x0c51] has been reserved
[    0.487233] system 00:01: [io  0x0c52] has been reserved
[    0.487234] system 00:01: [io  0x0c6c] has been reserved
[    0.487236] system 00:01: [io  0x0c6f] has been reserved
[    0.487237] system 00:01: [io  0x0cd0-0x0cd1] has been reserved
[    0.487239] system 00:01: [io  0x0cd2-0x0cd3] has been reserved
[    0.487240] system 00:01: [io  0x0cd4-0x0cd5] has been reserved
[    0.487242] system 00:01: [io  0x0cd6-0x0cd7] has been reserved
[    0.487243] system 00:01: [io  0x0cd8-0x0cdf] has been reserved
[    0.487245] system 00:01: [io  0x0800-0x089f] has been reserved
[    0.487247] system 00:01: [io  0x0b20-0x0b3f] has been reserved
[    0.487250] system 00:01: [io  0x0900-0x090f] has been reserved
[    0.487252] system 00:01: [io  0x0910-0x091f] has been reserved
[    0.487253] system 00:01: [io  0xfe00-0xfefe] has been reserved
[    0.487256] system 00:01: [mem 0xfec00000-0xfec00fff] could not be reserved
[    0.487258] system 00:01: [mem 0xfee00000-0xfee00fff] has been reserved
[    0.487259] system 00:01: [mem 0xfed80000-0xfed8ffff] has been reserved
[    0.487261] system 00:01: [mem 0xfed61000-0xfed70fff] has been reserved
[    0.487263] system 00:01: [mem 0xfec10000-0xfec10fff] has been reserved
[    0.487265] system 00:01: [mem 0xfed00000-0xfed00fff] has been reserved
[    0.487266] system 00:01: [mem 0xffc00000-0xffffffff] has been reserved
[    0.487269] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487372] system 00:02: [io  0x0290-0x02af] has been reserved
[    0.487374] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487385] pnp 00:03: [dma 4]
[    0.487401] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
[    0.487432] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.487450] pnp 00:05: Plug and Play ACPI device, IDs PNP0800 (active)
[    0.487498] system 00:06: [io  0x04d0-0x04d1] has been reserved
[    0.487500] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487524] pnp 00:07: Plug and Play ACPI device, IDs PNP0c04 (active)
[    0.487553] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487671] system 00:09: [mem 0xfeb20000-0xfeb23fff] could not be reserved
[    0.487673] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487750] system 00:0a: [mem 0xfec20000-0xfec200ff] could not be reserved
[    0.487752] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.487889] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active)
[    0.487894] pnp: PnP ACPI: found 12 devices
[    0.487895] ACPI: bus type PNP unregistered
[    0.494204] pci 0000:00:06.0: BAR 15: assigned [mem 0xd0900000-0xd09fffff pref]
[    0.494209] pci 0000:00:0a.0: BAR 15: assigned [mem 0xd0a00000-0xd0afffff pref]
[    0.494211] pci 0000:00:02.0: PCI bridge to [bus 01]
[    0.494214] pci 0000:00:02.0:   bridge window [io  0xe000-0xefff]
[    0.494217] pci 0000:00:02.0:   bridge window [mem 0xfea00000-0xfeafffff]
[    0.494219] pci 0000:00:02.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.494222] pci 0000:00:04.0: PCI bridge to [bus 02]
[    0.494223] pci 0000:00:04.0:   bridge window [io  0xd000-0xdfff]
[    0.494227] pci 0000:00:04.0:   bridge window [mem 0xd0800000-0xd08fffff 64bit pref]
[    0.494230] pci 0000:03:00.0: BAR 6: assigned [mem 0xd0900000-0xd090ffff pref]
[    0.494231] pci 0000:00:06.0: PCI bridge to [bus 03]
[    0.494233] pci 0000:00:06.0:   bridge window [io  0xc000-0xcfff]
[    0.494236] pci 0000:00:06.0:   bridge window [mem 0xfe900000-0xfe9fffff]
[    0.494238] pci 0000:00:06.0:   bridge window [mem 0xd0900000-0xd09fffff pref]
[    0.494241] pci 0000:04:00.0: BAR 6: assigned [mem 0xd0a00000-0xd0a0ffff pref]
[    0.494242] pci 0000:00:0a.0: PCI bridge to [bus 04]
[    0.494244] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    0.494246] pci 0000:00:0a.0:   bridge window [mem 0xfe800000-0xfe8fffff]
[    0.494281] pci 0000:00:0a.0:   bridge window [mem 0xd0a00000-0xd0afffff pref]
[    0.494284] pci 0000:00:14.4: PCI bridge to [bus 05]
[    0.494288] pci 0000:00:14.4:   bridge window [mem 0xfe700000-0xfe7fffff]
[    0.494291] pci 0000:00:14.4:   bridge window [mem 0xd0000000-0xd07fffff pref]
[    0.494499] pci_bus 0000:00: resource 4 [io  0x0000-0x03af]
[    0.494500] pci_bus 0000:00: resource 5 [io  0x03e0-0x0cf7]
[    0.494502] pci_bus 0000:00: resource 6 [io  0x03b0-0x03df]
[    0.494503] pci_bus 0000:00: resource 7 [io  0x0d00-0xffff]
[    0.494505] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff]
[    0.494507] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff]
[    0.494508] pci_bus 0000:00: resource 10 [mem 0xc0000000-0xffffffff]
[    0.494510] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    0.494511] pci_bus 0000:01: resource 1 [mem 0xfea00000-0xfeafffff]
[    0.494513] pci_bus 0000:01: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.494515] pci_bus 0000:02: resource 0 [io  0xd000-0xdfff]
[    0.494516] pci_bus 0000:02: resource 2 [mem 0xd0800000-0xd08fffff 64bit pref]
[    0.494518] pci_bus 0000:03: resource 0 [io  0xc000-0xcfff]
[    0.494520] pci_bus 0000:03: resource 1 [mem 0xfe900000-0xfe9fffff]
[    0.494521] pci_bus 0000:03: resource 2 [mem 0xd0900000-0xd09fffff pref]
[    0.494523] pci_bus 0000:04: resource 0 [io  0xb000-0xbfff]
[    0.494524] pci_bus 0000:04: resource 1 [mem 0xfe800000-0xfe8fffff]
[    0.494526] pci_bus 0000:04: resource 2 [mem 0xd0a00000-0xd0afffff pref]
[    0.494527] pci_bus 0000:05: resource 1 [mem 0xfe700000-0xfe7fffff]
[    0.494529] pci_bus 0000:05: resource 2 [mem 0xd0000000-0xd07fffff pref]
[    0.494531] pci_bus 0000:05: resource 4 [io  0x0000-0x03af]
[    0.494532] pci_bus 0000:05: resource 5 [io  0x03e0-0x0cf7]
[    0.494533] pci_bus 0000:05: resource 6 [io  0x03b0-0x03df]
[    0.494535] pci_bus 0000:05: resource 7 [io  0x0d00-0xffff]
[    0.494536] pci_bus 0000:05: resource 8 [mem 0x000a0000-0x000bffff]
[    0.494538] pci_bus 0000:05: resource 9 [mem 0x000c0000-0x000dffff]
[    0.494539] pci_bus 0000:05: resource 10 [mem 0xc0000000-0xffffffff]
[    0.494576] NET: Registered protocol family 2
[    0.494761] TCP established hash table entries: 32768 (order: 7, 524288 bytes)
[    0.494924] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
[    0.495078] TCP: Hash tables configured (established 32768 bind 32768)
[    0.495131] TCP: reno registered
[    0.495141] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    0.495165] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    0.495264] NET: Registered protocol family 1
[    1.555947] pci 0000:05:05.0: Boot video device
[    1.555953] PCI: CLS 64 bytes, default 64
[    1.555998] Unpacking initramfs...
[    1.600551] Freeing initrd memory: 2864k freed
[    1.601842] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.601843] AMD-Vi: Interrupt remapping enabled
[    1.601858] pci 0000:00:00.2: irq 72 for MSI/MSI-X
[    1.602091] AMD-Vi: Initialized for Passthrough Mode
[    1.684133] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.684135] software IO TLB [mem 0xb9a5f000-0xbda5f000] (64MB) mapped at [ffff8800b9a5f000-ffff8800bda5efff]
[    1.684267] LVT offset 1 assigned for vector 0x400
[    1.684271] IBS: LVT offset 1 assigned
[    1.684303] perf: AMD IBS detected (0x0000001f)
[    1.684326] Scanning for low memory corruption every 60 seconds
[    1.684502] audit: initializing netlink socket (disabled)
[    1.684512] type=2000 audit(1375822060.419:1): initialized
[    1.693786] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.695305] VFS: Disk quotas dquot_6.5.2
[    1.695355] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.695547] msgmni has been set to 7837
[    1.695958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    1.696084] io scheduler noop registered
[    1.696087] io scheduler deadline registered
[    1.696119] io scheduler cfq registered (default)
[    1.696535] vesafb: mode is 800x600x32, linelength=3200, pages=0
[    1.696538] vesafb: scrolling: redraw
[    1.696539] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.696642] vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90010800000, using 1920k, total 1920k
[    1.836131] Console: switching to colour frame buffer device 100x37
[    1.974614] fb0: VESA VGA frame buffer device
[    1.974645] GHES: HEST is not enabled!
[    1.974720] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.975117] Linux agpgart interface v0.103
[    1.975160] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.975567] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.975592] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.975820] mousedev: PS/2 mouse device common for all mice
[    1.975864] rtc_cmos 00:04: RTC can wake from S4
[    1.975992] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    1.976016] rtc_cmos 00:04: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    1.976024] cpuidle: using governor ladder
[    1.976025] cpuidle: using governor menu
[    1.976077] drop_monitor: Initializing network drop monitor service
[    1.976134] TCP: cubic registered
[    1.976209] NET: Registered protocol family 10
[    1.976352] NET: Registered protocol family 17
[    1.976361] Key type dns_resolver registered
[    1.976651] PM: Hibernation image not present or could not be loaded.
[    1.976659] registered taskstats version 1
[    1.977428]   Magic number: 9:819:800
[    1.977528] rtc_cmos 00:04: setting system clock to 2013-08-06 20:47:41 UTC (1375822061)
[    1.978344] Freeing unused kernel memory: 1124k freed
[    1.978626] Write protecting the kernel read-only data: 8192k
[    1.981160] Freeing unused kernel memory: 1212k freed
[    1.982425] Freeing unused kernel memory: 408k freed
[    1.997013] systemd-udevd[58]: starting version 204
[    1.998130] pci-stub: add 1002:6739 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.998486] pci-stub 0000:01:00.0: claimed by stub
[    1.998548] pci-stub: add 1002:AA88 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.998590] pci-stub 0000:01:00.1: claimed by stub
[    2.024612] SCSI subsystem initialized
[    2.024825] ACPI: bus type USB registered
[    2.024864] usbcore: registered new interface driver usbfs
[    2.024877] usbcore: registered new interface driver hub
[    2.028146] usbcore: registered new device driver usb
[    2.028278] ACPI: bus type ATA registered
[    2.028894] libata version 3.00 loaded.
[    2.028976] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.029187] ehci-pci: EHCI PCI platform driver
[    2.029391] ehci-pci 0000:00:12.2: EHCI Host Controller
[    2.029402] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.029407] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 1
[    2.029425] QUIRK: Enable AMD PLL fix
[    2.029428] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.029440] ehci-pci 0000:00:12.2: debug port 1
[    2.029492] ehci-pci 0000:00:12.2: irq 17, io mem 0xfeb09000
[    2.038798] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00
[    2.038974] hub 1-0:1.0: USB hub found
[    2.038978] hub 1-0:1.0: 5 ports detected
[    2.039226] ehci-pci 0000:00:13.2: EHCI Host Controller
[    2.039231] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 2
[    2.039234] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.039243] ehci-pci 0000:00:13.2: debug port 1
[    2.039276] ehci-pci 0000:00:13.2: irq 21, io mem 0xfeb07000
[    2.048873] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00
[    2.049091] hub 2-0:1.0: USB hub found
[    2.049097] hub 2-0:1.0: 5 ports detected
[    2.049357] ehci-pci 0000:00:16.2: EHCI Host Controller
[    2.049363] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 3
[    2.049366] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[    2.049375] ehci-pci 0000:00:16.2: debug port 1
[    2.049409] ehci-pci 0000:00:16.2: irq 23, io mem 0xfeb04000
[    2.058786] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00
[    2.058998] hub 3-0:1.0: USB hub found
[    2.059003] hub 3-0:1.0: 4 ports detected
[    2.059231] ohci_hcd 0000:00:12.0: OHCI Host Controller
[    2.059237] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 4
[    2.059270] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfeb0a000
[    2.116269] hub 4-0:1.0: USB hub found
[    2.116276] hub 4-0:1.0: 5 ports detected
[    2.116497] ohci_hcd 0000:00:13.0: OHCI Host Controller
[    2.116502] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 5
[    2.116528] ohci_hcd 0000:00:13.0: irq 20, io mem 0xfeb08000
[    2.172906] hub 5-0:1.0: USB hub found
[    2.172913] hub 5-0:1.0: 5 ports detected
[    2.173134] ohci_hcd 0000:00:14.5: OHCI Host Controller
[    2.173139] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 6
[    2.173154] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfeb06000
[    2.229540] hub 6-0:1.0: USB hub found
[    2.229547] hub 6-0:1.0: 2 ports detected
[    2.229744] ohci_hcd 0000:00:16.0: OHCI Host Controller
[    2.229749] ohci_hcd 0000:00:16.0: new USB bus registered, assigned bus number 7
[    2.229804] ohci_hcd 0000:00:16.0: irq 22, io mem 0xfeb05000
[    2.286169] hub 7-0:1.0: USB hub found
[    2.286177] hub 7-0:1.0: 4 ports detected
[    2.286369] ahci 0000:00:11.0: version 3.0
[    2.286604] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    2.286607] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
[    2.288415] scsi0 : ahci
[    2.288607] scsi1 : ahci
[    2.288821] scsi2 : ahci
[    2.289018] scsi3 : ahci
[    2.289198] scsi4 : ahci
[    2.289383] scsi5 : ahci
[    2.289485] ata1: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b100 irq 19
[    2.289487] ata2: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b180 irq 19
[    2.289490] ata3: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b200 irq 19
[    2.289492] ata4: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b280 irq 19
[    2.289494] ata5: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b300 irq 19
[    2.289496] ata6: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b380 irq 19
[    2.289724] ahci 0000:03:00.0: AHCI 0001.0100 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
[    2.289727] ahci 0000:03:00.0: flags: 64bit ncq pm led clo pmp pio slum part 
[    2.290127] scsi6 : ahci
[    2.290308] scsi7 : ahci
[    2.290403] ata7: SATA max UDMA/133 abar m512@0xfe910000 port 0xfe910100 irq 51
[    2.290406] ata8: SATA max UDMA/133 abar m512@0xfe910000 port 0xfe910180 irq 51
[    2.290610] ahci 0000:04:00.0: AHCI 0001.0100 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
[    2.290613] ahci 0000:04:00.0: flags: 64bit ncq pm led clo pmp pio slum part 
[    2.291008] scsi8 : ahci
[    2.291214] scsi9 : ahci
[    2.291310] ata9: SATA max UDMA/133 abar m512@0xfe810000 port 0xfe810100 irq 47
[    2.291313] ata10: SATA max UDMA/133 abar m512@0xfe810000 port 0xfe810180 irq 47
[    2.608518] ata9: SATA link down (SStatus 0 SControl 300)
[    2.608552] ata7: SATA link down (SStatus 0 SControl 300)
[    2.608560] ata2: SATA link down (SStatus 0 SControl 300)
[    2.608604] ata5: SATA link down (SStatus 0 SControl 300)
[    2.608606] ata10: SATA link down (SStatus 0 SControl 300)
[    2.608653] ata8: SATA link down (SStatus 0 SControl 300)
[    2.608654] ata4: SATA link down (SStatus 0 SControl 300)
[    2.608685] ata6: SATA link down (SStatus 0 SControl 300)
[    2.608713] ata3: SATA link down (SStatus 0 SControl 300)
[    2.685092] tsc: Refined TSC clocksource calibration: 3010.011 MHz
[    2.685104] Switching to clocksource tsc
[    2.765042] usb 7-4: new full-speed USB device number 2 using ohci_hcd
[    2.775047] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[    2.775798] ata1.00: ATA-8: ST9160827AS, 3.AAA, max UDMA/133
[    2.775801] ata1.00: 312581808 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.776269] ata1.00: configured for UDMA/133
[    2.776476] scsi 0:0:0:0: Direct-Access     ATA      ST9160827AS      3.AA PQ: 0 ANSI: 5
[    2.777492] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB)
[    2.777643] sd 0:0:0:0: [sda] Write Protect is off
[    2.777645] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.777684] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.836861]  sda: sda1 sda2 sda3 sda4
[    2.837752] sd 0:0:0:0: [sda] Attached SCSI disk
[    2.934093] hidraw: raw HID events driver (C) Jiri Kosina
[    2.954737] usbcore: registered new interface driver usbhid
[    2.954741] usbhid: USB HID core driver
[    2.955529] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.0/input/input0
[    2.955580] hid-generic 0003:045E:0745.0001: input,hidraw0: USB HID v1.11 Keyboard [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input0
[    2.956007] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.1/input/input1
[    2.956144] hid-generic 0003:045E:0745.0002: input,hidraw1: USB HID v1.11 Mouse [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input1
[    2.970799] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.2/input/input2
[    2.970969] hid-generic 0003:045E:0745.0003: input,hiddev0,hidraw2: USB HID v1.11 Device [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input2
[    4.117161] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[    5.056668] systemd[1]: systemd 204 running in system mode. (+PAM -LIBWRAP -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
[    5.100219] systemd[1]: Set hostname to <Virtking>.
[    5.778615] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory. See system logs and 'systemctl status display-manager.service' for details.
[    5.778815] systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
[    5.778859] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    5.778870] systemd[1]: Expecting device sys-subsystem-net-devices-eth0.device...
[    5.781751] systemd[1]: Starting Remote File Systems.
[    5.784017] systemd[1]: Reached target Remote File Systems.
[    5.784026] systemd[1]: Starting LVM2 metadata daemon socket.
[    5.786533] systemd[1]: Listening on LVM2 metadata daemon socket.
[    5.786541] systemd[1]: Starting /dev/initctl Compatibility Named Pipe.
[    5.789397] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    5.789404] systemd[1]: Starting Device-mapper event daemon FIFOs.
[    5.792079] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    5.792087] systemd[1]: Starting Delayed Shutdown Socket.
[    5.794432] systemd[1]: Listening on Delayed Shutdown Socket.
[    5.794462] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
[    5.798466] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    5.798531] systemd[1]: Starting udev Kernel Socket.
[    5.800678] systemd[1]: Listening on udev Kernel Socket.
[    5.800714] systemd[1]: Starting udev Control Socket.
[    5.802918] systemd[1]: Listening on udev Control Socket.
[    5.802926] systemd[1]: Starting Encrypted Volumes.
[    5.805116] systemd[1]: Reached target Encrypted Volumes.
[    5.805129] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
[    5.805161] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    5.805168] systemd[1]: Starting Paths.
[    5.806901] systemd[1]: Reached target Paths.
[    5.806914] systemd[1]: Starting Journal Socket.
[    5.808928] systemd[1]: Listening on Journal Socket.
[    5.838913] systemd[1]: Started Set Up Additional Binary Formats.
[    5.838933] systemd[1]: Starting Setup Virtual Console...
[    5.841446] systemd[1]: Starting udev Coldplug all Devices...
[    5.844029] systemd[1]: Starting Create static device nodes in /dev...
[    5.853689] systemd[1]: Starting Apply Kernel Variables...
[    5.856159] systemd[1]: Mounting POSIX Message Queue File System...
[    5.859606] systemd[1]: Starting Load Kernel Modules...
[    5.921078] systemd[1]: Mounting Debug File System...
[    5.983231] systemd[1]: Starting Journal Service...
[    6.105438] systemd[1]: Started Journal Service.
[    6.105490] systemd[1]: Mounting Huge Pages File System...
[    6.167710] systemd[1]: Expecting device dev-sda3.device...
[    6.227811] systemd[1]: Starting File System Check on Root Device...
[    6.287283] systemd[1]: Mounting Temporary Directory...
[    6.534911] systemd[1]: Expecting device dev-sda1.device...
[    6.648418] systemd[1]: Started Create static device nodes in /dev.
[    6.704478] systemd[1]: Started Apply Kernel Variables.
[    6.760296] systemd[1]: Mounted POSIX Message Queue File System.
[    6.815358] systemd[1]: Mounted Debug File System.
[    6.869511] systemd[1]: Mounted Huge Pages File System.
[    6.921472] systemd[1]: Mounted Temporary Directory.
[    6.972157] systemd[1]: Started udev Coldplug all Devices.
[    6.972232] systemd[1]: Starting udev Kernel Device Manager...
[    7.158863] systemd-udevd[157]: starting version 204
[    7.453239] kvm: Nested Virtualization enabled
[    7.453244] kvm: Nested Paging enabled
[    7.956172] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input3
[    7.956179] ACPI: Power Button [PWRB]
[    7.962263] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input4
[    7.962270] ACPI: Power Button [PWRF]
[    7.964106] wmi: Mapper loaded
[    8.038713] EXT4-fs (sda2): re-mounted. Opts: data=ordered
[    8.214618] microcode: CPU0: patch_level=0x010000c8
[    8.493444] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    8.493726] r8169 0000:02:00.0: irq 73 for MSI/MSI-X
[    8.493898] r8169 0000:02:00.0 eth0: RTL8168evl/8111evl at 0xffffc90011292000, c8:60:00:85:03:7f, XID 0c900800 IRQ 73
[    8.493901] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[    8.549878] MCE: In-kernel MCE decoding enabled.
[    8.558059] EDAC MC: Ver: 3.0.0
[    8.559106] AMD64 EDAC driver v3.4.0
[    8.559163] EDAC amd64: DRAM ECC disabled.
[    8.559181] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
 Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
 (Note that use of the override may cause unknown side effects.)
[    8.808637] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    8.885809] input: HDA ATI SB Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card0/input5
[    8.885839] input: HDA ATI SB Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card0/input6
[    8.885860] input: HDA ATI SB Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card0/input7
[    8.885883] input: HDA ATI SB Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card0/input8
[    8.885906] input: HDA ATI SB Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card0/input9
[    8.885930] input: HDA ATI SB Line as /devices/pci0000:00/0000:00:14.2/sound/card0/input10
[    8.885959] input: HDA ATI SB Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input11
[    8.885981] input: HDA ATI SB Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input12
[    8.906550] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[    8.963091] input: PC Speaker as /devices/platform/pcspkr/input/input13
[    9.093876] microcode: CPU1: patch_level=0x010000c8
[    9.093902] microcode: CPU2: patch_level=0x010000c8
[    9.093920] microcode: CPU3: patch_level=0x010000c8
[    9.094006] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    9.189679] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05
[    9.189726] sp5100_tco: PCI Revision ID: 0x42
[    9.189759] sp5100_tco: Using 0xfed80b00 for watchdog MMIO address
[    9.189774] sp5100_tco: Last reboot was not triggered by watchdog.
[    9.189875] sp5100_tco: initialized (0xffffc90012458b00). heartbeat=60 sec (nowayout=0)
[    9.250142] acpi-cpufreq: overriding BIOS provided _PSD data
[    9.309658] asus_wmi: ASUS WMI generic driver loaded
[    9.347460] asus_wmi: Initialization: 0x0
[    9.347523] asus_wmi: BIOS WMI version: 0.9
[    9.347637] asus_wmi: SFUN value: 0x0
[    9.348546] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input14
[    9.350939] asus_wmi: Disabling ACPI video driver
[   10.188668] Adding 8388604k swap on /dev/sda3.  Priority:-1 extents:1 across:8388604k FS
[   10.502171] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[   10.525220] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
[   11.634768] it87: Found IT8721F chip at 0x290, revision 1
[   12.172445] r8169 0000:02:00.0 eth0: link down
[   12.172480] r8169 0000:02:00.0 eth0: link down
[   12.172533] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   14.096652] r8169 0000:02:00.0 eth0: link up
[   14.096672] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   15.830233] ip_tables: (C) 2000-2006 Netfilter Core Team
[   15.875583] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   26.940720] Bridge firewalling registered
[   27.204314] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   27.267605] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[  146.061275] VFIO - User Level meta-driver version: 0.3
[  158.100190] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

Last edited by Diggo (2013-08-06 18:01:19)

Offline

#417 2013-08-06 21:54:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:

So, today i took a chance to migrate to vfio-pci. But i got stuck at the point, where to start the DomU the first time. If i read the posts here right, the patched versions are for VGA-reset only (so for testing, i actually don´t need them?).

You need them

It also looks like your trying to passthrough your primary vga device, that wont work, the SDL error is because it doesnt have permission to access the X server, as a regular user try:

xhost +

Last edited by nbhs (2013-08-06 22:01:51)

Offline

#418 2013-08-06 22:07:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

licious wrote:

Specs :
cpu:Intel i7 3770
motherboard : Asus P8H77-I
host gpu : Intel HD 4000
guest gpu : Nvidia GT 620
Host : Ubuntu 13.04 with patched kernel 3.10.1
grub: GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=8086:0151,10de:0f01,10de:0bea vfio_iommu_type1.allow_unsafe_interrupts=1"
guest : Ubuntu 13.04, kernel 3.8.0-27-generic (stock)


I've built the patched kernel, seabios and qemu - but I can't get an output on the screen connected to the GT620 when I use vfio. Qemu output is always blank / black as well. If I use "-device pci-assign" instead of "-device vfio-pci" then my guest os initializes the GT620 when starting X on the guest (no output at all before X starts). I can only get an output if I'm using the Nouveau-drivers on the guest, Nvidia binary drivers won't work. I've tried patching the drivers as well, but I've had no luck in getting the patched drivers to work. The output from dmesg on the guest :
[   17.260770] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[   17.260778] NVRM: rm_init_adapter(0) failed

the weird thing is that it seems like hdmi sound is initialized :
[   13.078985] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input5
[   13.079086] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input6
[   13.079149] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input7
[   13.079225] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input8

Also, when using -device vfio-pci I get an error message saying "qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/xbmc/GT620.rom: Warning, device 0000:01:00.0 does not support reset". This message does not appear when I use pci-assign, most likely because no kernel modules have been loaded for the GPU yet. I thought the patched kernel and/ or qemu would take care of resetting the device as well ?

From previous posts it looks like the intel driver also causes problems. Try blacklisting it and check if you get output with vfio

Offline

#419 2013-08-07 09:42:55

Diggo
Member
Registered: 2013-05-29
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

It also looks like your trying to passthrough your primary vga device, that wont work, the SDL error is because it doesnt have permission to access the X server, as a regular user try:

xhost +

Sorry, i had the primary card cut out. It is a oldschool Matrox Millenium smile

05:05.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 2064W [Millennium] (rev 01)

I had no X server In the last setup -> I installed it and the xorg-xhost too, but still noch change. I will now try your patched kernels.

*edit*

Found the following in this thread smile

nbhs wrote:

Qemu by default creates a sdl window, if you want to lauch qemu without X launch qemu with this option

-nographic

Seems like it is doing something now smile I can´t say what, because i am not at home and i just try this over SSH wink ... but hope rises again big_smile
Thanks for all your support so far!

Last edited by Diggo (2013-08-07 10:31:45)

Offline

#420 2013-08-07 20:55:18

Diggo
Member
Registered: 2013-05-29
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm... beeing home a while and playing around with this, i installed the x server, some lxde for googling and so son ... qemu now starts without any SDL Error (inside a x session), but its just giving me a new blank window.  At least qemu seems to be running now.
Sadly the output on the tv-screen (where the passed through card is connected via hdmi) is still just black. I don´t see the seabios messages. Removing "-vga none" shows the seabios then in the mentioned qemu window.
So i guess qemu is running into seabios, but i can´t see it on the passed-through card.

Is there any more debug-information i can grab?

*update* Got it! Looked like something in the patched qemu got corrupted. Redownloaded nbhs´ packages and everything went on smoothly wink

Last edited by Diggo (2013-08-07 22:52:10)

Offline

#421 2013-08-11 03:07:21

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'll start saying that I'm trying this with gentoo so I'm restricted to certain package versions; please don't scorn me.

I'm using a 3.10.5 kernel patched with:
- kernel-vfio-vga-reset.patch.tar.gz
- pci: Enable overrides for missing ACS capabilities

QEMU 1.5.2 patched with:
- qemu-vfio-vga-reset.patch.tar.gz

seabios 1.7.2.2 patched with:
- pciinit: Enable default VGA device


The cards:

# lspci
[...]
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks PRO [Radeon HD 6570]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Turks/Whistler HDMI Audio [Radeon HD 6000 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV740 PRO [Radeon HD 4770]
02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV710/730 HDMI Audio [Radeon HD 4000 series]
03:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cedar [Radeon HD 5000/6000/7350 Series]
03:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cedar HDMI Audio [Radeon HD 5400/6300 Series]
# lspci -n
[...]
01:00.0 0300: 1002:6759
01:00.1 0403: 1002:aa90
02:00.0 0300: 1002:94b3
02:00.1 0403: 1002:aa38
03:00.0 0300: 1002:68f9
03:00.1 0403: 1002:aa68

Kernel options:

# cat .config | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
# cat .config | grep IOMMU
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
# CONFIG_AMD_IOMMU is not set
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
# cat .config | grep CONFIG_CMDLINE
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="root=/dev/sda2 pci-stub.ids=1002:94b3,1002:aa38,1002:68f9,1002:aa68 pcie_acs_override=downstream kvm_intel.emulate_invalid_guest_state=0 vfio_iommu_type1.allow_unsafe_interrupts=1"
# CONFIG_CMDLINE_OVERRIDE is not set

My vfio-bind (I use this because all the cards used to be on the same IOMMU group, and even though now it's not the case anymore the result is the same):

#!/bin/bash

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

The cards get properly binded:

# ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
lrwxrwxrwx 1 root root    0 Aug 11 05:02 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root    0 Aug 11 05:02 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1
lrwxrwxrwx 1 root root    0 Aug 11 05:02 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:03:00.0
lrwxrwxrwx 1 root root    0 Aug 11 05:02 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:03:00.1
--w------- 1 root root 4096 Aug 11 05:02 bind
lrwxrwxrwx 1 root root    0 Aug 11 05:02 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 Aug 11 04:51 new_id
--w------- 1 root root 4096 Aug 11 05:02 remove_id
--w------- 1 root root 4096 Aug 11 04:51 uevent
--w------- 1 root root 4096 Aug 11 05:02 unbind

dmesg when I start the test with qemu:

[   33.741710] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[   33.926077] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
[...]
[  195.150275] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[  195.337386] vfio-pci 0000:02:00.1: enabling device (0000 -> 0002)

Result after all this: the screen stays black.

I tried swapping the cards and the displays, and nothing changes.
I want to think that I'm overlooking something simple. I just hope that is the case.

Well, one thing that bothers me is that some time after boot vgaarb does this:

[   22.719722] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[   22.719724] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:03:00.0

But I don't know how relevant it can be. 02:00.0 doesn't work either after all.


EDIT:

I think I'm getting closer: I tried booting archlinux-2013.08.01-dual.iso instead of just testing the bios screen. The display was still black, but I tried grabbing the input and hitting enter and after a few seconds I saw the prompt and it worked.

Now, there are still a few problems:
- If I halt the guest system, qemu doesn't close itself, and even after I forcefully kill it the last image still stays in the monitor and doesn't go away until I re-run qemu. I'm guessing this means the card isn't getting reset?
- I also tried a Windows installation disc, but with that the display stayed black and when I tried to re-run the archlinux CD I got a kernel panic after a while (drm_kms_helper: panic occurred, switching back to text console). From then on the only way to get it to work again is to reboot the host.

Meanwhile dmesg... (2 "good" runs, 1 try with the Windows CD and the last one that ended with a kernel panic)

[   32.052373] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[   32.237194] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
[   37.622600] kvm [2066]: vcpu0 unhandled rdmsr: 0x345
[   39.575702] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[   41.154701] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X
[  134.870611] kvm [2082]: vcpu0 unhandled rdmsr: 0x345
[  136.815406] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[  138.310509] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X
[  183.069869] kvm [2093]: vcpu0 unhandled rdmsr: 0x345
[  185.015290] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[  186.511395] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X
[  360.835017] kvm [2163]: vcpu0 unhandled rdmsr: 0x345
[  362.786883] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[  364.322938] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X

EDIT2:

Just tried to install an archlinux guest. Managed to complete the installation but when I rebooted the VM I couldn't get any kind of output from the display connected to the card. Switching to the cirrus card I could see that everything was fine and the card was there when I did lspci.
I also tried dumping the bioses of all my cards (got to mount them one at at time on the first PCIe slot since I couldn't manage to get the roms otherwise, damn Input/Output errors) and to run it with them. Nothing changed.


EDIT3:

Oh, I think I may have got it.
Changing the framebuffer device with "fbcon=map:1" lets the kernel use the right one.
So this means there is some other framebuffer device that gets priority during the boot process, namely:

fb0: EFI VGA frame buffer device

Is this part of the Q35? Time to investigate.

Last edited by Evonat (2013-08-12 05:05:34)

Offline

#422 2013-08-11 22:30:00

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hopefully we'll have this last issue snubbed out eventually. I'm good on non-vfio-pci, but vfio still blue-screens on that Radeon driver.

If this helps, I noticed also that the usb PCI device I pass through is also incredibly slow. As in, my mouse movement is very... choppy? Jerky? And the USB hotspot I hooked up worked but downloaded things VERY slowly (devices hooked up to the same spot via Wi-Fi did not have this issue). My hopes are that it's a software bug somewhere down the line, and possibly is related to having 000:05+ pci devices breaking iommu.

Offline

#423 2013-08-12 05:05:46

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to try installing Windows 8, but since I couldn't manage to use the right card from the boot, I decided to try installing it with the cirrus and switching to the ATI afterwards, leaving the PCIe devices there (1), or just removing them and adding them after the installation is complete (2), or just hot plugging them once the system is fully installed (3).
Sure enough, all three failed:
BSOD with:

VIDEO_TDR_FAILURE (atikmpag.sys)

Now I'll try with pci-assign, which I wanted to avoid, but I'm losing hope.
It's quite disappointing since I built this system specifically with this in mind, and even though it's still quite immature it seemed the "cleanest" solution and I wanted to avoid the bloaded xen and vmware.


EDIT:

Almost forgot to say, the screen turned on before the BSOD, and stayed on even afterwards. On, and black.


EDIT2:

Well that worked. Well, almost: the jerky mouse makes the system unusable. The keyboard works fine and the animations seem fluid enough. Just the damn mouse. I had the same problem with the cirrus card, plus a constant "white refresh" in the the VM window (I don't know how to explain it... have you even seen what happens when you point an analog video camera to a CRT display? That.)
I even though that the problem was that I still had the VM window output enabled, so I tried "-vga none", but the screen stayed off, probably Windows refuses to boot because it can't find the primary graphic adapter or something like that; I also thought of "-nographic", but I don't know how can I grab mouse and keyboard after I start it like that.

Last edited by Evonat (2013-08-12 05:30:31)

Offline

#424 2013-08-12 14:46:47

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm trying to make it work with an archlinux guest. It seems to work fine. That is, until I decide that I want X:

X.Org X Server 1.14.2
Release Date: 2013-06-25
[    99.537] X Protocol Version 11, Revision 0
[    99.539] Build Operating System: Linux 3.9.7-1-ARCH x86_64 
[    99.541] Current Operating System: Linux Arch 3.10.5-1-ARCH #1 SMP PREEMPT Mon Aug 5 08:04:22 CEST 2013 x86_64
[    99.541] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=75e17ae6-53fe-42ec-9ef6-a144c6fe404a rw fbcon=map:1 quiet
[    99.544] Build Date: 01 July 2013  10:48:42AM
[    99.546]  
[    99.548] Current version of pixman: 0.30.2
[    99.552] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[    99.552] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[    99.560] (==) Log file: "/var/log/Xorg.0.log", Time: Mon Aug 12 16:28:20 2013
[    99.563] (==) Using config directory: "/etc/X11/xorg.conf.d"
[    99.565] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[    99.565] (==) No Layout section.  Using the first Screen section.
[    99.565] (==) No screen section available. Using defaults.
[    99.565] (**) |-->Screen "Default Screen Section" (0)
[    99.565] (**) |   |-->Monitor "<default monitor>"
[    99.565] (==) No monitor specified for screen "Default Screen Section".
	Using a default monitor configuration.
[    99.565] (==) Automatically adding devices
[    99.565] (==) Automatically enabling devices
[    99.565] (==) Automatically adding GPU devices
[    99.565] (WW) The directory "/usr/share/fonts/OTF/" does not exist.
[    99.565] 	Entry deleted from font path.
[    99.565] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/100dpi/".
[    99.565] 	Entry deleted from font path.
[    99.565] 	(Run 'mkfontdir' on "/usr/share/fonts/100dpi/").
[    99.565] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/75dpi/".
[    99.565] 	Entry deleted from font path.
[    99.565] 	(Run 'mkfontdir' on "/usr/share/fonts/75dpi/").
[    99.565] (==) FontPath set to:
	/usr/share/fonts/misc/,
	/usr/share/fonts/TTF/,
	/usr/share/fonts/Type1/
[    99.565] (==) ModulePath set to "/usr/lib/xorg/modules"
[    99.565] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[    99.565] (II) Loader magic: 0x7fdc20
[    99.565] (II) Module ABI versions:
[    99.565] 	X.Org ANSI C Emulation: 0.4
[    99.565] 	X.Org Video Driver: 14.1
[    99.565] 	X.Org XInput driver : 19.1
[    99.565] 	X.Org Server Extension : 7.0
[    99.565] (II) xfree86: Adding drm device (/dev/dri/card0)
[    99.567] (--) PCI:*(0:1:0:0) 1002:68f9:174b:e233 rev 0, Mem @ 0xe0000000/268435456, 0xfea00000/131072, I/O @ 0x0000c000/256, BIOS @ 0x????????/131072
[    99.569] Initializing built-in extension Generic Event Extension
[    99.571] Initializing built-in extension SHAPE
[    99.573] Initializing built-in extension MIT-SHM
[    99.575] Initializing built-in extension XInputExtension
[    99.577] Initializing built-in extension XTEST
[    99.579] Initializing built-in extension BIG-REQUESTS
[    99.581] Initializing built-in extension SYNC
[    99.582] Initializing built-in extension XKEYBOARD
[    99.584] Initializing built-in extension XC-MISC
[    99.586] Initializing built-in extension SECURITY
[    99.588] Initializing built-in extension XINERAMA
[    99.589] Initializing built-in extension XFIXES
[    99.591] Initializing built-in extension RENDER
[    99.592] Initializing built-in extension RANDR
[    99.594] Initializing built-in extension COMPOSITE
[    99.596] Initializing built-in extension DAMAGE
[    99.597] Initializing built-in extension MIT-SCREEN-SAVER
[    99.599] Initializing built-in extension DOUBLE-BUFFER
[    99.601] Initializing built-in extension RECORD
[    99.602] Initializing built-in extension DPMS
[    99.604] Initializing built-in extension X-Resource
[    99.606] Initializing built-in extension XVideo
[    99.607] Initializing built-in extension XVideo-MotionCompensation
[    99.609] Initializing built-in extension XFree86-VidModeExtension
[    99.611] Initializing built-in extension XFree86-DGA
[    99.612] Initializing built-in extension XFree86-DRI
[    99.614] Initializing built-in extension DRI2
[    99.614] (II) "glx" will be loaded by default.
[    99.614] (II) LoadModule: "dri2"
[    99.614] (II) Module "dri2" already built-in
[    99.614] (II) LoadModule: "glamoregl"
[    99.614] (II) Loading /usr/lib/xorg/modules/libglamoregl.so
[    99.615] (II) Module glamoregl: vendor="X.Org Foundation"
[    99.615] 	compiled for 1.14.0, module version = 0.5.0
[    99.615] 	ABI class: X.Org ANSI C Emulation, version 0.4
[    99.615] (II) LoadModule: "glx"
[    99.615] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[    99.615] (II) Module glx: vendor="X.Org Foundation"
[    99.615] 	compiled for 1.14.2, module version = 1.0.0
[    99.615] 	ABI class: X.Org Server Extension, version 7.0
[    99.615] (==) AIGLX enabled
[    99.617] Loading extension GLX
[    99.617] (==) Matched ati as autoconfigured driver 0
[    99.617] (==) Matched ati as autoconfigured driver 1
[    99.617] (==) Matched vesa as autoconfigured driver 2
[    99.617] (==) Matched modesetting as autoconfigured driver 3
[    99.617] (==) Matched fbdev as autoconfigured driver 4
[    99.617] (==) Assigned the driver to the xf86ConfigLayout
[    99.617] (II) LoadModule: "ati"
[    99.617] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so
[    99.617] (II) Module ati: vendor="X.Org Foundation"
[    99.617] 	compiled for 1.14.2, module version = 7.2.0
[    99.617] 	Module class: X.Org Video Driver
[    99.617] 	ABI class: X.Org Video Driver, version 14.1
[    99.617] (II) LoadModule: "radeon"
[    99.617] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so
[    99.617] (II) Module radeon: vendor="X.Org Foundation"
[    99.617] 	compiled for 1.14.2, module version = 7.2.0
[    99.617] 	Module class: X.Org Video Driver
[    99.617] 	ABI class: X.Org Video Driver, version 14.1
[    99.617] (II) LoadModule: "vesa"
[    99.617] (WW) Warning, couldn't open module vesa
[    99.617] (II) UnloadModule: "vesa"
[    99.617] (II) Unloading vesa
[    99.617] (EE) Failed to load module "vesa" (module does not exist, 0)
[    99.617] (II) LoadModule: "modesetting"
[    99.617] (WW) Warning, couldn't open module modesetting
[    99.617] (II) UnloadModule: "modesetting"
[    99.617] (II) Unloading modesetting
[    99.617] (EE) Failed to load module "modesetting" (module does not exist, 0)
[    99.617] (II) LoadModule: "fbdev"
[    99.617] (WW) Warning, couldn't open module fbdev
[    99.617] (II) UnloadModule: "fbdev"
[    99.617] (II) Unloading fbdev
[    99.617] (EE) Failed to load module "fbdev" (module does not exist, 0)
[    99.617] (II) RADEON: Driver for ATI Radeon chipsets:
	ATI Radeon Mobility X600 (M24) 3150 (PCIE), ATI FireMV 2400 (PCI),
	ATI Radeon Mobility X300 (M24) 3152 (PCIE),
	ATI FireGL M24 GL 3154 (PCIE), ATI FireMV 2400 3155 (PCI),
	ATI Radeon X600 (RV380) 3E50 (PCIE),
	ATI FireGL V3200 (RV380) 3E54 (PCIE), ATI Radeon IGP320 (A3) 4136,
	ATI Radeon IGP330/340/350 (A4) 4137, ATI Radeon 9500 AD (AGP),
	ATI Radeon 9500 AE (AGP), ATI Radeon 9600TX AF (AGP),
	ATI FireGL Z1 AG (AGP), ATI Radeon 9800SE AH (AGP),
	ATI Radeon 9800 AI (AGP), ATI Radeon 9800 AJ (AGP),
	ATI FireGL X2 AK (AGP), ATI Radeon 9600 AP (AGP),
	ATI Radeon 9600SE AQ (AGP), ATI Radeon 9600XT AR (AGP),
	ATI Radeon 9600 AS (AGP), ATI FireGL T2 AT (AGP), ATI Radeon 9650,
	ATI FireGL RV360 AV (AGP), ATI Radeon 7000 IGP (A4+) 4237,
	ATI Radeon 8500 AIW BB (AGP), ATI Radeon IGP320M (U1) 4336,
	ATI Radeon IGP330M/340M/350M (U2) 4337,
	ATI Radeon Mobility 7000 IGP 4437, ATI Radeon 9000/PRO If (AGP/PCI),
	ATI Radeon 9000 Ig (AGP/PCI), ATI Radeon X800 (R420) JH (AGP),
	ATI Radeon X800PRO (R420) JI (AGP),
	ATI Radeon X800SE (R420) JJ (AGP), ATI Radeon X800 (R420) JK (AGP),
	ATI Radeon X800 (R420) JL (AGP), ATI FireGL X3 (R420) JM (AGP),
	ATI Radeon Mobility 9800 (M18) JN (AGP),
	ATI Radeon X800 SE (R420) (AGP), ATI Radeon X800XT (R420) JP (AGP),
	ATI Radeon X800 VE (R420) JT (AGP), ATI Radeon X850 (R480) (AGP),
	ATI Radeon X850 XT (R480) (AGP), ATI Radeon X850 SE (R480) (AGP),
	ATI Radeon X850 PRO (R480) (AGP), ATI Radeon X850 XT PE (R480) (AGP),
	ATI Radeon Mobility M7 LW (AGP),
	ATI Mobility FireGL 7800 M7 LX (AGP),
	ATI Radeon Mobility M6 LY (AGP), ATI Radeon Mobility M6 LZ (AGP),
	ATI FireGL Mobility 9000 (M9) Ld (AGP),
	ATI Radeon Mobility 9000 (M9) Lf (AGP),
	ATI Radeon Mobility 9000 (M9) Lg (AGP), ATI FireMV 2400 PCI,
	ATI Radeon 9700 Pro ND (AGP), ATI Radeon 9700/9500Pro NE (AGP),
	ATI Radeon 9600TX NF (AGP), ATI FireGL X1 NG (AGP),
	ATI Radeon 9800PRO NH (AGP), ATI Radeon 9800 NI (AGP),
	ATI FireGL X2 NK (AGP), ATI Radeon 9800XT NJ (AGP),
	ATI Radeon Mobility 9600/9700 (M10/M11) NP (AGP),
	ATI Radeon Mobility 9600 (M10) NQ (AGP),
	ATI Radeon Mobility 9600 (M11) NR (AGP),
	ATI Radeon Mobility 9600 (M10) NS (AGP),
	ATI FireGL Mobility T2 (M10) NT (AGP),
	ATI FireGL Mobility T2e (M11) NV (AGP), ATI Radeon QD (AGP),
	ATI Radeon QE (AGP), ATI Radeon QF (AGP), ATI Radeon QG (AGP),
	ATI FireGL 8700/8800 QH (AGP), ATI Radeon 8500 QL (AGP),
	ATI Radeon 9100 QM (AGP), ATI Radeon 7500 QW (AGP/PCI),
	ATI Radeon 7500 QX (AGP/PCI), ATI Radeon VE/7000 QY (AGP/PCI),
	ATI Radeon VE/7000 QZ (AGP/PCI), ATI ES1000 515E (PCI),
	ATI Radeon Mobility X300 (M22) 5460 (PCIE),
	ATI Radeon Mobility X600 SE (M24C) 5462 (PCIE),
	ATI FireGL M22 GL 5464 (PCIE), ATI Radeon X800 (R423) UH (PCIE),
	ATI Radeon X800PRO (R423) UI (PCIE),
	ATI Radeon X800LE (R423) UJ (PCIE),
	ATI Radeon X800SE (R423) UK (PCIE),
	ATI Radeon X800 XTP (R430) (PCIE), ATI Radeon X800 XL (R430) (PCIE),
	ATI Radeon X800 SE (R430) (PCIE), ATI Radeon X800 (R430) (PCIE),
	ATI FireGL V7100 (R423) (PCIE), ATI FireGL V5100 (R423) UQ (PCIE),
	ATI FireGL unknown (R423) UR (PCIE),
	ATI FireGL unknown (R423) UT (PCIE),
	ATI Mobility FireGL V5000 (M26) (PCIE),
	ATI Mobility FireGL V5000 (M26) (PCIE),
	ATI Mobility Radeon X700 XL (M26) (PCIE),
	ATI Mobility Radeon X700 (M26) (PCIE),
	ATI Mobility Radeon X700 (M26) (PCIE),
	ATI Radeon X550XTX 5657 (PCIE), ATI Radeon 9100 IGP (A5) 5834,
	ATI Radeon Mobility 9100 IGP (U3) 5835,
	ATI Radeon XPRESS 200 5954 (PCIE),
	ATI Radeon XPRESS 200M 5955 (PCIE), ATI Radeon 9250 5960 (AGP),
	ATI Radeon 9200 5961 (AGP), ATI Radeon 9200 5962 (AGP),
	ATI Radeon 9200SE 5964 (AGP), ATI FireMV 2200 (PCI),
	ATI ES1000 5969 (PCI), ATI Radeon XPRESS 200 5974 (PCIE),
	ATI Radeon XPRESS 200M 5975 (PCIE),
	ATI Radeon XPRESS 200 5A41 (PCIE),
	ATI Radeon XPRESS 200M 5A42 (PCIE),
	ATI Radeon XPRESS 200 5A61 (PCIE),
	ATI Radeon XPRESS 200M 5A62 (PCIE),
	ATI Radeon X300 (RV370) 5B60 (PCIE),
	ATI Radeon X600 (RV370) 5B62 (PCIE),
	ATI Radeon X550 (RV370) 5B63 (PCIE),
	ATI FireGL V3100 (RV370) 5B64 (PCIE),
	ATI FireMV 2200 PCIE (RV370) 5B65 (PCIE),
	ATI Radeon Mobility 9200 (M9+) 5C61 (AGP),
	ATI Radeon Mobility 9200 (M9+) 5C63 (AGP),
	ATI Mobility Radeon X800 XT (M28) (PCIE),
	ATI Mobility FireGL V5100 (M28) (PCIE),
	ATI Mobility Radeon X800 (M28) (PCIE), ATI Radeon X850 5D4C (PCIE),
	ATI Radeon X850 XT PE (R480) (PCIE),
	ATI Radeon X850 SE (R480) (PCIE), ATI Radeon X850 PRO (R480) (PCIE),
	ATI unknown Radeon / FireGL (R480) 5D50 (PCIE),
	ATI Radeon X850 XT (R480) (PCIE),
	ATI Radeon X800XT (R423) 5D57 (PCIE),
	ATI FireGL V5000 (RV410) (PCIE), ATI Radeon X700 XT (RV410) (PCIE),
	ATI Radeon X700 PRO (RV410) (PCIE),
	ATI Radeon X700 SE (RV410) (PCIE), ATI Radeon X700 (RV410) (PCIE),
	ATI Radeon X700 SE (RV410) (PCIE), ATI Radeon X1800,
	ATI Mobility Radeon X1800 XT, ATI Mobility Radeon X1800,
	ATI Mobility FireGL V7200, ATI FireGL V7200, ATI FireGL V5300,
	ATI Mobility FireGL V7100, ATI Radeon X1800, ATI Radeon X1800,
	ATI Radeon X1800, ATI Radeon X1800, ATI Radeon X1800,
	ATI FireGL V7300, ATI FireGL V7350, ATI Radeon X1600, ATI RV505,
	ATI Radeon X1300/X1550, ATI Radeon X1550, ATI M54-GL,
	ATI Mobility Radeon X1400, ATI Radeon X1300/X1550,
	ATI Radeon X1550 64-bit, ATI Mobility Radeon X1300,
	ATI Mobility Radeon X1300, ATI Mobility Radeon X1300,
	ATI Mobility Radeon X1300, ATI Radeon X1300, ATI Radeon X1300,
	ATI RV505, ATI RV505, ATI FireGL V3300, ATI FireGL V3350,
	ATI Radeon X1300, ATI Radeon X1550 64-bit, ATI Radeon X1300/X1550,
	ATI Radeon X1600, ATI Radeon X1300/X1550, ATI Mobility Radeon X1450,
	ATI Radeon X1300/X1550, ATI Mobility Radeon X2300,
	ATI Mobility Radeon X2300, ATI Mobility Radeon X1350,
	ATI Mobility Radeon X1350, ATI Mobility Radeon X1450,
	ATI Radeon X1300, ATI Radeon X1550, ATI Mobility Radeon X1350,
	ATI FireMV 2250, ATI Radeon X1550 64-bit, ATI Radeon X1600,
	ATI Radeon X1650, ATI Radeon X1600, ATI Radeon X1600,
	ATI Mobility FireGL V5200, ATI Mobility Radeon X1600,
	ATI Radeon X1650, ATI Radeon X1650, ATI Radeon X1600,
	ATI Radeon X1300 XT/X1600 Pro, ATI FireGL V3400,
	ATI Mobility FireGL V5250, ATI Mobility Radeon X1700,
	ATI Mobility Radeon X1700 XT, ATI FireGL V5200,
	ATI Mobility Radeon X1700, ATI Radeon X2300HD,
	ATI Mobility Radeon HD 2300, ATI Mobility Radeon HD 2300,
	ATI Radeon X1950, ATI Radeon X1900, ATI Radeon X1950,
	ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900,
	ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900,
	ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900,
	ATI AMD Stream Processor, ATI Radeon X1900, ATI Radeon X1950,
	ATI RV560, ATI RV560, ATI Mobility Radeon X1900, ATI RV560,
	ATI Radeon X1950 GT, ATI RV570, ATI RV570, ATI FireGL V7400,
	ATI RV560, ATI Radeon X1650, ATI Radeon X1650, ATI RV560,
	ATI Radeon 9100 PRO IGP 7834, ATI Radeon Mobility 9200 IGP 7835,
	ATI Radeon X1200, ATI Radeon X1200, ATI Radeon X1200,
	ATI Radeon X1200, ATI Radeon X1200, ATI RS740, ATI RS740M, ATI RS740,
	ATI RS740M, ATI Radeon HD 2900 XT, ATI Radeon HD 2900 XT,
	ATI Radeon HD 2900 XT, ATI Radeon HD 2900 Pro, ATI Radeon HD 2900 GT,
	ATI FireGL V8650, ATI FireGL V8600, ATI FireGL V7600,
	ATI Radeon 4800 Series, ATI Radeon HD 4870 x2,
	ATI Radeon 4800 Series, ATI Radeon HD 4850 x2,
	ATI FirePro V8750 (FireGL), ATI FirePro V7760 (FireGL),
	ATI Mobility RADEON HD 4850, ATI Mobility RADEON HD 4850 X2,
	ATI Radeon 4800 Series, ATI FirePro RV770, AMD FireStream 9270,
	AMD FireStream 9250, ATI FirePro V8700 (FireGL),
	ATI Mobility RADEON HD 4870, ATI Mobility RADEON M98,
	ATI Mobility RADEON HD 4870, ATI Radeon 4800 Series,
	ATI Radeon 4800 Series, ATI FirePro M7750, ATI M98, ATI M98, ATI M98,
	ATI Mobility Radeon HD 4650, ATI Radeon RV730 (AGP),
	ATI Mobility Radeon HD 4670, ATI FirePro M5750,
	ATI Mobility Radeon HD 4670, ATI Radeon RV730 (AGP),
	ATI RV730XT [Radeon HD 4670], ATI RADEON E4600,
	ATI Radeon HD 4600 Series, ATI RV730 PRO [Radeon HD 4650],
	ATI FirePro V7750 (FireGL), ATI FirePro V5700 (FireGL),
	ATI FirePro V3750 (FireGL), ATI Mobility Radeon HD 4830,
	ATI Mobility Radeon HD 4850, ATI FirePro M7740, ATI RV740,
	ATI Radeon HD 4770, ATI Radeon HD 4700 Series, ATI Radeon HD 4770,
	ATI FirePro M5750, ATI RV610, ATI Radeon HD 2400 XT,
	ATI Radeon HD 2400 Pro, ATI Radeon HD 2400 PRO AGP, ATI FireGL V4000,
	ATI RV610, ATI Radeon HD 2350, ATI Mobility Radeon HD 2400 XT,
	ATI Mobility Radeon HD 2400, ATI RADEON E2400, ATI RV610,
	ATI FireMV 2260, ATI RV670, ATI Radeon HD3870,
	ATI Mobility Radeon HD 3850, ATI Radeon HD3850,
	ATI Mobility Radeon HD 3850 X2, ATI RV670,
	ATI Mobility Radeon HD 3870, ATI Mobility Radeon HD 3870 X2,
	ATI Radeon HD3870 X2, ATI FireGL V7700, ATI Radeon HD3850,
	ATI Radeon HD3690, AMD Firestream 9170, ATI Radeon HD 4550,
	ATI Radeon RV710, ATI Radeon RV710, ATI Radeon RV710,
	ATI Radeon HD 4350, ATI Mobility Radeon 4300 Series,
	ATI Mobility Radeon 4500 Series, ATI Mobility Radeon 4500 Series,
	ATI FirePro RG220, ATI Mobility Radeon 4330, ATI RV630,
	ATI Mobility Radeon HD 2600, ATI Mobility Radeon HD 2600 XT,
	ATI Radeon HD 2600 XT AGP, ATI Radeon HD 2600 Pro AGP,
	ATI Radeon HD 2600 XT, ATI Radeon HD 2600 Pro, ATI Gemini RV630,
	ATI Gemini Mobility Radeon HD 2600 XT, ATI FireGL V5600,
	ATI FireGL V3600, ATI Radeon HD 2600 LE,
	ATI Mobility FireGL Graphics Processor, ATI Radeon HD 3470,
	ATI Mobility Radeon HD 3430, ATI Mobility Radeon HD 3400 Series,
	ATI Radeon HD 3450, ATI Radeon HD 3450, ATI Radeon HD 3430,
	ATI Radeon HD 3450, ATI FirePro V3700, ATI FireMV 2450,
	ATI FireMV 2260, ATI FireMV 2260, ATI Radeon HD 3600 Series,
	ATI Radeon HD 3650 AGP, ATI Radeon HD 3600 PRO,
	ATI Radeon HD 3600 XT, ATI Radeon HD 3600 PRO,
	ATI Mobility Radeon HD 3650, ATI Mobility Radeon HD 3670,
	ATI Mobility FireGL V5700, ATI Mobility FireGL V5725,
	ATI Radeon HD 3200 Graphics, ATI Radeon 3100 Graphics,
	ATI Radeon HD 3200 Graphics, ATI Radeon 3100 Graphics,
	ATI Radeon HD 3300 Graphics, ATI Radeon HD 3200 Graphics,
	ATI Radeon 3000 Graphics, SUMO, SUMO, SUMO2, SUMO2, SUMO2, SUMO2,
	SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, ATI Radeon HD 4200,
	ATI Radeon 4100, ATI Mobility Radeon HD 4200,
	ATI Mobility Radeon 4100, ATI Radeon HD 4290, ATI Radeon HD 4250,
	AMD Radeon HD 6310 Graphics, AMD Radeon HD 6310 Graphics,
	AMD Radeon HD 6250 Graphics, AMD Radeon HD 6250 Graphics,
	AMD Radeon HD 6300 Series Graphics,
	AMD Radeon HD 6200 Series Graphics, PALM, PALM, PALM, CYPRESS,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI FirePro (FireGL) Graphics Adapter, AMD Firestream 9370,
	AMD Firestream 9350, ATI Radeon HD 5800 Series,
	ATI Radeon HD 5800 Series, ATI Radeon HD 5800 Series,
	ATI Radeon HD 5800 Series, ATI Radeon HD 5900 Series,
	ATI Radeon HD 5900 Series, ATI Mobility Radeon HD 5800 Series,
	ATI Mobility Radeon HD 5800 Series,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI Mobility Radeon HD 5800 Series, ATI Radeon HD 5700 Series,
	ATI Radeon HD 5700 Series, ATI Radeon HD 6700 Series,
	ATI Radeon HD 5700 Series, ATI Radeon HD 6700 Series,
	ATI Mobility Radeon HD 5000 Series,
	ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon HD 5570,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI FirePro (FireGL) Graphics Adapter, ATI Radeon HD 5670,
	ATI Radeon HD 5570, ATI Radeon HD 5500 Series, REDWOOD,
	ATI Mobility Radeon HD 5000 Series,
	ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon Graphics,
	ATI Mobility Radeon Graphics, CEDAR,
	ATI FirePro (FireGL) Graphics Adapter,
	ATI FirePro (FireGL) Graphics Adapter, ATI FirePro 2270, CEDAR,
	ATI Radeon HD 5450, CEDAR, CEDAR, CAYMAN, CAYMAN, CAYMAN, CAYMAN,
	CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN,
	AMD Radeon HD 6900 Series, AMD Radeon HD 6900 Series, CAYMAN, CAYMAN,
	CAYMAN, AMD Radeon HD 6900M Series, Mobility Radeon HD 6000 Series,
	BARTS, BARTS, Mobility Radeon HD 6000 Series,
	Mobility Radeon HD 6000 Series, BARTS, BARTS, BARTS, BARTS,
	AMD Radeon HD 6800 Series, AMD Radeon HD 6800 Series,
	AMD Radeon HD 6700 Series, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS,
	TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS,
	TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS,
	CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS,
	CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, ARUBA, ARUBA,
	ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA,
	ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA,
	ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA,
	ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, TAHITI, TAHITI, TAHITI, TAHITI,
	TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI,
	TAHITI, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN,
	PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN,
	VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE,
	VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE,
	VERDE, VERDE, VERDE, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND,
	OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, HAINAN, HAINAN, HAINAN,
	HAINAN, HAINAN, HAINAN, BONAIRE, BONAIRE, BONAIRE, BONAIRE, BONAIRE,
	BONAIRE, BONAIRE, BONAIRE, KABINI, KABINI, KABINI, KABINI, KABINI,
	KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI,
	KABINI, KABINI, KABINI
[    99.620] (--) using VT number 2

[    99.631] (II) [KMS] Kernel modesetting enabled.
[    99.631] (EE) No devices detected.
[    99.631] (EE) 
Fatal server error:
[    99.631] (EE) no screens found(EE) 
[    99.631] (EE) 
Please consult the The X.Org Foundation support 
	 at http://wiki.x.org
 for help. 
[    99.631] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[    99.631] (EE) 

It looks like it can't identify the card, despite detecting it and loading the right drivers:

[    99.567] (--) PCI:*(0:1:0:0) 1002:68f9:174b:e233 rev 0, Mem @ 0xe0000000/268435456, 0xfea00000/131072, I/O @ 0x0000c000/256, BIOS @ 0x????????/131072
[    99.617] (II) RADEON: Driver for ATI Radeon chipsets:
[...]

Really, I don't get it.


EDIT:

Oh, that wasn't hard, I just needed to create a small xorg config file with this to use the right bus:

Section "Device"
    Identifier "Card0"
    Driver "radeon"
    BusID "PCI:1:0:0"
EndSection

The mouse is still unusable though. I think I'll connect some old mouse for the host and I'll passthrough the current one hotplugging it on occurrence on various guests.

Last edited by Evonat (2013-08-12 15:10:36)

Offline

#425 2013-08-14 23:25:50

irtimmer
Member
Registered: 2013-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have try this on my new computer (Nvdia GTX 660 and Intel Hasswell 4600). I have patched seabios, qemu and kernel mentioned in the opening post. It works while using the ArchLinux cd but I can't get it work under Seabios and Windows. It looks like there is still something missing in Seabios to properly initialize the GPU. Using ArchLinux to visualize Linux isn't very useful for me so I hope someone knows how to fix this. I could of course provide some more debug information if someone can tell me what is useful.

While running qemu with -vga none it is by default not possible to grab the keyboard and mouse in the Qemu window. After searching for an option to enable this while not using a emulated video card driver I have found somekind of a hack. If you add '-device qxl' to the Qemu command you would get a second video card which is showed in the Qemu window. In my testing it looks it won't cause conflicts (probably because it doesn't have a video bios).

Offline

Board footer

Powered by FluxBB