You are not logged in.

#26 2015-11-04 20:57:18

letni69
Member
Registered: 2015-09-21
Posts: 17

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

schefi wrote:

Today I compiled and checked with 4.4.0-rc0 from fedora rawhide and unmodified 4.3 stable (directly from kernel.org). Issue is still present both cases.
Some of you reported the guest working after a few minutes of slow period, thoug I can't seem to boot the guest at all, even if waiting for like 20 minutes. An MCE error is also reported after about 1 minute. But the whole stuff works fine with 4.1.7 kernel.
I can't put my finger on it. PCI passthrough related MCE error, and slowdown bugs are present in kernel bugzilla tickets, and 4.2.4 (and some other) changelogs claim to have reverted corresponding changes, because of the very same symptoms we experience. If they have been reverted, then how come it still doesn't work in 4.3?

Probably because the reverted changes did not affected your bug you experienced. You should probably open your own bug report.

Offline

#27 2015-11-06 21:34:07

jimrif
Member
Registered: 2015-11-06
Posts: 2

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

I too am having issues with kernels above 4.1 with PCI passthrough, however I have found that lowering the amount of ram for the VM to around 2.5G would allow the VM to run normally.  It would be interesting to note if less ram helps some of the issues others are seeing.

Offline

#28 2015-11-07 10:25:01

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

jimrif wrote:

I too am having issues with kernels above 4.1 with PCI passthrough, however I have found that lowering the amount of ram for the VM to around 2.5G would allow the VM to run normally

I can confirm. Originally guest had 24GB of memory, changing it to 2.4 GB allows guest to run normal with 4.3 kernel. Increasing memory to 3GB results the same slowness/lockup as discussed earlier. Jimrif's observation could proove useful in debugging the issue.

Offline

#29 2015-11-07 17:14:44

Cadeyrn
Member
Registered: 2013-04-06
Posts: 170

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

It's probably the case for me, too. I give my VMs 8GB because they're for gaming.

Offline

#30 2015-11-08 18:02:12

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

I finally managed to make time for a git bisect. There were more than 1000 commits to verify, took about 14 rounds to narrow it down. It turns out, there was a set of commits in 4.1-rc2 regarding kvm and mtrr. These are in mainline since 4.2.

To be very specific, bisect lead to the following: commit fa61213746a706dd975661151c35795ca4dd82c2 KVM: MTRR: simplify kvm_mtrr_get_guest_memory_type. The introduction of this patch broke PCI passthrough for me. It is related to another patch in this set: introduce mtrr_for_each_mem_type. I should examine the changes in the source code, although I'm affraid I'm not qualified for this. I'll let you know if I find something,. Maybe all this is already enough for filing a decent bug report.

Another thing I noticed while doing the iterations of the bisect. At some point I was presented with a bug, I had not seen before. Maybe it relates to the delays some of you mentioned earlier. The startup was delayed about a minute or so. I mean when starting the guest I had to wait 1-2 minutes for the screen to turn on, (no sync signal at all.) Then magically the EDK II EFI environment showed up, and booting went normal from there on. I think this bug got eventually fixed in some 4.2 version of the kernel, I didn't experience anything like this with stock fedora kernels. I could live with the delay, but more importantly the main bug is still in mainline. I can't boot up the guest with 4.2 and newer kernels, if guest memory is larger than 2.4-2.5 gigabytes.

Offline

#31 2015-11-09 12:21:13

delta_50
Member
Registered: 2015-11-09
Posts: 1

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Offline

#32 2015-11-09 22:08:24

Thralas
Member
Registered: 2007-06-26
Posts: 32

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

hiliev wrote:
Cadeyrn wrote:

That's interesting. I'm still on linux-lts 4.1.9-2 and it works fine. I'll try 10 and see what happens later. But, I thought the linux-vfio kernel is meant for VMs that don't use UEFI to have a much easier time with OVMF.

The problem that I experience with linux-lts 4.1.10-1 is that the VM hangs/spins right at the TianoCore splash screen. This is different than the 4.2.x problem where the VM boot is extremely slow. I will continue tracking the official LTS kernel and see what happens in the future, but until the problem is resolved, I'm sticking to the vfio version from AUR. Also, I'm getting a ~3 fps boost in The Witcher 3 since switching from 4.1.6-1 to the latest linux-vfio-lts, but that could easily be attributed to the new Nvidia driver as well.

I also seem to be experiencing this issue, on a fresh install (so I havent' got PCIe passthrough to work reliably ever).

Passthrough initially works, but after ~1 minute my guest seems to freeze (in Windows), resuling in video output dropping and 1 core being pegged. Afterwards, all attempts at starting the guest result in the core being pegged straight away (doesn't even attempt to boot).

Tried all of linux, linux-lts and linux-vfio-lts to no avail. Guest currently has only 2 GB of memory assigned - doesn't help either.

So it seems there are 2 major bugs, the bugzilla posted above not covering my case?

Offline

#33 2015-11-16 17:53:49

dhiru1602
Member
Registered: 2015-01-28
Posts: 13

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

My system doesn't even boot on 4.2+ kernels with intel_iommu=on. It just freezes at "Loading initial ramdisk"

Offline

#34 2015-11-21 10:11:42

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Made some progress on narrowing down the failing code part. There is a temporary fix, you only have to modify one line kernel source. For debugging purposes I modified the kvm_mtrr_get_guest_memory_type function to always return MTRR_TYPE_WRBACK.

That way guest boots correctly!

The lack of caching in guests MTRR could cause the extreme slowness of VMs. I also posted some details on Bugzilla. https://bugzilla.kernel.org/show_bug.cgi?id=107561

Offline

#35 2015-11-21 17:07:54

letni69
Member
Registered: 2015-09-21
Posts: 17

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

schefi wrote:

Made some progress on narrowing down the failing code part. There is a temporary fix, you only have to modify one line kernel source. For debugging purposes I modified the kvm_mtrr_get_guest_memory_type function to always return MTRR_TYPE_WRBACK.

That way guest boots correctly!

The lack of caching in guests MTRR could cause the extreme slowness of VMs. I also posted some details on Bugzilla. https://bugzilla.kernel.org/show_bug.cgi?id=107561

Another way to debug this issue is to print out the return value of the kvm_mtrr_get_guest_memory_type function so we can compare what return value difference we will get when comparing both revisions of the code.
You can try something like this:
http://pastebin.com/G0FbW9Hh
Just do the same with the 4.1 kernel code (adding the printk function calls) and you should see with dmesg the return values of the functions.

Offline

#36 2015-11-23 19:42:39

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

letni69 wrote:

You can try something like this:
http://pastebin.com/G0FbW9Hh

I did as you suggested. Though I didn't put the printk in kvm_mtrr_get_guest_memory_type(), beaucause there isn't such a function in 4.1. It was introduced in 4.2. It is only called from vmx_get_mt_mask(), so i put it there instead, and looked for the return value, along with the same arguments you suggested.
I already uploaded the result for a 4.2.6 kernel with 2GB memory guest and a 3GB memory guest to shared storage. I also did a trace as Paolo Bonzini suggested. More details: https://bugzilla.kernel.org/show_bug.cgi?id=107561
The printk from 4.1 is still left todo. I'll do it in a few days.

Offline

#37 2015-11-25 02:05:11

frostbittenking
Member
Registered: 2013-05-01
Posts: 52

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Sigh, I think this also hit linux-lts 4.1.12 and upwards (don't know about the versions between 4.1.7 and 4.1.12). Tried to build a custom Windows 8.1 for a friend for his new computer in qemu, and the installation is slower than my grandmother with her rollator (if I still had one). Either stuck in the first step (copying windows files), or at preparing files for installation .. , usually at 0%, sometimes it gets to 1%, but then it just does nothing, although mouse is movable and it let's me even abort the installation without delay.
Usually this step takes between 2 and 4 minutes in qemu.

Offline

#38 2015-11-25 02:44:39

Cadeyrn
Member
Registered: 2013-04-06
Posts: 170

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

^Doubt it. I'm currently running lts 4.1.13 and still have no issues with my Windows VM.

Offline

#39 2015-11-25 10:06:04

frostbittenking
Member
Registered: 2013-05-01
Posts: 52

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

I don't know, it's the only explanation. Did you install it in 4.1.13? Don't ask me, what the problem exactly is, but running it (tested some others I had already installed) is no problem. Only the installation (for me) is a problem. I let the installation run last night, and it stopped with some files not beeing found - which is not true, since the installation worked with the same iso and the same qcow2 in VirtualBox. Something is seriously flaky here. Only way to try is to build an old kernel from the aur or manual. Idk.

Offline

#40 2015-11-25 13:17:23

frostbittenking
Member
Registered: 2013-05-01
Posts: 52

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Damn. Red Herring. Probably has something to do with this flaky shitty intenso Disk. Don't know why, but qemu is doing worse compared to virtualbox or direct operations on the disk.
EDIT: Might be the kernel version after all(4.1.6 and 8gb ram worked without problems .. tested it with 2.5GB ram according to a suggestion above, but that worked flaky too).

Last edited by frostbittenking (2015-11-25 14:56:01)

Offline

#41 2015-11-25 17:13:42

Cadeyrn
Member
Registered: 2013-04-06
Posts: 170

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

And you were passing through a PCI device in all of these tests? I'm not sure you have the same problem as us, because for us, all VMs running all OSes have the issue (not just the Winodws installer), whether they were created before or after the upgrade.

Offline

#42 2015-11-25 17:23:54

frostbittenking
Member
Registered: 2013-05-01
Posts: 52

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

As I said, I can't be sure, it looks like the symptom described (the slowdown), but it seems very specific in my case, and rolling back to 4.1.6 seems to have solved that problem. I wasn't able to install in my case win 8.1 in kernel 4.1.13 or 4.1.12 in qemu. Unfortunately, bisecting this problem takes more time than I can sacrifice. For now, I stay with 4.1.6 during the qemu stuff.

Offline

#43 2015-11-25 20:00:41

Thralas
Member
Registered: 2007-06-26
Posts: 32

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Supposedly this is the fix for the slowdown a fix for OVMF slowdows (see below): [PATCH] KVM: x86: obey KVM_X86_QUIRK_CD_NW_CLEARED in kvm_set_cr0().

My R9 380 still crashes as soon asn Windows attempts to load the AMD (non-vesa) driver. Doubt the above helps in my cases. Curious to hear who's still having issues with the patch applied.

Last edited by Thralas (2015-11-26 21:23:15)

Offline

#44 2015-11-25 21:16:14

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Sorry, but this is a no go.
I just compiled a 4.4-rc1 kernel. I checked, this patch is already included. And Windows guest boot issue is still the same. After a long wait with logo the white dots start move slowly, than it seems to do it endlessly.
If I recall correctly this patch is for the 1-2 minute delay on starting the guest. But that bug only occured on guest startup. It took a long time before EDKII was ready, than after that everything went fine. It think this patch is for that former bug, not for wrong cache types with vMTRR.
Nevertheless I tried it, just to be sure, and it didn't work. Still looking for a solution.

Offline

#45 2015-12-16 17:10:43

kgadek
Member
Registered: 2015-12-16
Posts: 2

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Anyone working on this / around this issue in any way?

Offline

#46 2015-12-16 17:17:21

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

There is a kernel bug ticket in progress. https://bugzilla.kernel.org/show_bug.cgi?id=107561
Most of the news can be found there.

Offline

#47 2015-12-23 16:00:44

schefi
Member
Registered: 2015-10-27
Posts: 10

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

I inform you (in case someone wouldn't follow the kernel bug ticket), that a breakthrough has been reached. To resolve the issue of guest memory size dependent slowdown with PCI passthrough on 4.2 and onward kernels, the following two patches have to be applied to your source tree. I checked it, it works, others confirmed too. If you'r interested in the details, follow up on the bugzilla conversation. https://bugzilla.kernel.org/show_bug.cgi?id=107561


diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c
index 9e8bf13572e6..666f09e48ada 100644
--- a/arch/x86/kvm/mtrr.c
+++ b/arch/x86/kvm/mtrr.c
@@ -300,7 +300,6 @@ static void var_mtrr_range(struct kvm_mtrr_range *range, u64 *start, u64 *end)
    *start = range->base & PAGE_MASK;

    mask = range->mask & PAGE_MASK;
-    mask |= ~0ULL << boot_cpu_data.x86_phys_bits;

    /* This cannot overflow because writing to the reserved bits of
     * variable MTRRs causes a #GP.
@@ -356,10 +355,14 @@ static void set_var_mtrr_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data)
    if (var_mtrr_range_is_valid(cur))
        list_del(&mtrr_state->var_ranges[index].node);

+    /* Extend the mask with all 1 bits to the left, since those
+     * bits must implicitly be 0.  The bits are then cleared
+     * when reading them.
+     */
    if (!is_mtrr_mask)
        cur->base = data;
    else
-        cur->mask = data;
+        cur->mask = data | (-1LL << cpuid_maxphyaddr(vcpu));

    /* add it to the list if it's enabled. */
    if (var_mtrr_range_is_valid(cur)) {
@@ -426,6 +429,8 @@ int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
            *pdata = vcpu->arch.mtrr_state.var_ranges[index].base;
        else
            *pdata = vcpu->arch.mtrr_state.var_ranges[index].mask;
+
+        *pdata &= (1ULL << cpuid_maxphyaddr(vcpu)) - 1;
    }

    return 0;



diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c
index 9e8bf13572e6..adc54e1d40a9 100644
--- a/arch/x86/kvm/mtrr.c
+++ b/arch/x86/kvm/mtrr.c
@@ -267,7 +267,7 @@ static int fixed_mtrr_addr_to_seg(u64 addr)

    for (seg = 0; seg < seg_num; seg++) {
        mtrr_seg = &fixed_seg_table[seg];
-        if (mtrr_seg->start >= addr && addr < mtrr_seg->end)
+        if (mtrr_seg->start <= addr && addr < mtrr_seg->end)
            return seg;
    }


I think, sooner or later theese changes will be included in the mainline kernel. With time everyones respective distro will have it in their repository. So this stands here only for thoose who  are impatient, or would like to test for themselves.

Offline

#48 2016-01-11 13:53:17

kgadek
Member
Registered: 2015-12-16
Posts: 2

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

Yesterday kernel 4.4 got released. Looks like it contains the patches so… yay! Did not try it yet.

Last edited by kgadek (2016-01-11 13:54:08)

Offline

#49 2016-01-22 05:17:06

Cadeyrn
Member
Registered: 2013-04-06
Posts: 170

Re: [SOLVED] linux 4.1.6-1 -> 4.2.2-1 breaks PCI passthrough in QEMU/KVM

I just installed it from the testing repo, and all is fixed for me. It also works completely fine. What a first impression from the testing repo.

Offline

Board footer

Powered by FluxBB