Arch Linux

Wooots

Member

Registered: 2015-01-29

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Installed Windows 10 TP on the weekend through virt-manager, edited xml (virsh-edit) to use ovmf. So now I can manage everything with libvirt that's really great.

One question, I had to deactivate the hv_* options (@aw thanks for your blog) to get Code 43 Error away from my Nvidia GTX 780 (Next time I'll consider buying an AMD GPU)
I've read that deactivating those extensions impose a performance penalty.

So has someone measured how big this penalty is?
Would be very interested in some numbers.

Last edited by Wooots (2015-02-03 18:01:33)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Installed Windows 10 TP on the weekend through virt-manager, edited xml (virsh-edit) to use ovmf. So now I can manage everything with libvirt that's really great.

One question, I had to deactivate the hv_* options (@aw thanks for your blog) to get Code 43 Error away from my Nvidia GTX 780 (Next time I'll consider buying an AMD GPU)
I've read that deactivating those extensions impose a performance penalty.

So has someone measured how big this penalty is?
Would be very interested in some numbers.

You can test this yourself, download a version of the nvidia driver prior to 337.88, re-enable all the hyperv extensions and run your benchmarks.  It depends on the benchmark what you'll see, but I'm sure folks here would appreciate seeing your results.  Note that you really want to test both hyperv enabled vs disabled on the same old driver, then for graphics, test again with the new driver and hyperv disabled.  I think you'll find that the new driver more than compensates for the missing hyperv extensions, but non-graphics related benchmarks may still suffer slightly.

Last edited by aw (2015-02-03 18:22:44)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is this impossible to achieve on Core 2 Duo P9500? I was unable to find precise information but one site claimed that VT-d is available only from i7 onwards.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is this impossible to achieve on Core 2 Duo P9500? I was unable to find precise information but one site claimed that VT-d is available only from i7 onwards.

Maybe.  For some reason Intel was shipping VT-d on laptop chipsets way before servers.  The Core 2 Duos were before Intel moved graphics onto the CPU.  With the graphics, VT-d also had to migrate to the CPU.  So in this generation, while the CPU doesn't show any signs of VT-d (http://ark.intel.com/products/35566), the chipset actually does (http://ark.intel.com/products/35135), assuming you're talking about the W500 in your sig and google told me the right chipset.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that's W500 with dual Intel+Ati graphics. Is there a way to check for sure whether it supports it?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that's W500 with dual Intel+Ati graphics. Is there a way to check for sure whether it supports it?

Boot with intel_iommu=on, run find /sys/kernel/iommu_groups  If you have groups and devices, then you have VT-d.  That of course doesn't guarantee that you'll be able to get VGA assignment working, hybrid graphic laptops can be tricky.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Wooots

Member

Registered: 2015-01-29

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok did a quick performance test

For the cpu-test I used the free geekbench and for gpu test I used 3D Mark Firestrike test.
My Hardware: i7-4930k, Geforce GTX-780

Results with hv options on:
  - Geekbench (Single): 3314
  - Geekbench (Multi): 20082
  - 3D Mark: 9424, Graphics Score 10441

Results with hv options off:
  - Geekbench (Single): 3203
  - Geekbench (Multi): 19410
  - 3D Mark: 9045, Graphics Score 10249

Cpufreq governor was set to ondemand, next time i'll use performance.
On the 3D Mark Score, the Graphics Score is more interesting, because I didn't install PhysX, and physics calculation has a high impact on 3D Mark Score (Correct me if I'm wrong).

To sum up:
CPU: -3.4% (96.6%)
GPU: -1.9% (98.1%)

It was just a quick test but for the first, hv options seems not a really big performance penalty. I can live with that.
I would have to do these performance tests multiple times to get a more accurate value. I guess the cpu value could be +-1% when tested multiple times.

Last edited by Wooots (2015-02-03 20:11:18)

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Boot with intel_iommu=on, run find /sys/kernel/iommu_groups  If you have groups and devices, then you have VT-d.  That of course doesn't guarantee that you'll be able to get VGA assignment working, hybrid graphic laptops can be tricky.

The groups and devices are there. I guess I will have to try, then.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, I didn't know that Intel had VT-D pre Core i7 processors. Super cool!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, I didn't know that Intel had VT-D pre Core i7 processors. Super cool!

It exists on many i5 processors as well.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

Last edited by Lockheed (2015-02-03 21:34:37)

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

Can you post your full 'lspci -nn' output?

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07)
00:01.0 PCI bridge [0604]: Intel Corporation Mobile 4 Series Chipset PCI Express Graphics Port [8086:2a41] (rev 07)
00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07)
00:03.0 Communication controller [0780]: Intel Corporation Mobile 4 Series Chipset MEI Controller [8086:2a44] (rev 07)
00:03.3 Serial controller [0700]: Intel Corporation Mobile 4 Series Chipset AMT SOL Redirection [8086:2a47] (rev 07)
00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 03)
00:1c.1 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 [8086:2942] (rev 03)
00:1c.2 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 [8086:2944] (rev 03)
00:1c.3 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 [8086:2946] (rev 03)
00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev 93)
00:1f.0 ISA bridge [0601]: Intel Corporation ICH9M-E LPC Interface Controller [8086:2917] (rev 03)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IBM/IEM (ICH9M/ICH9M-E) 4 port SATA Controller [AHCI mode] [8086:2929] (rev 03)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 03)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] RV635/M86 [Mobility Radeon HD 3650] [1002:9591]
03:00.0 Network controller [0280]: Intel Corporation Ultimate N WiFi Link 5300 [8086:4236]
04:00.0 Memory controller [0580]: Intel Corporation Turbo Memory Controller [8086:444e] (rev 11)
05:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
15:00.0 CardBus bridge [0607]: Ricoh Co Ltd RL5c476 II [1180:0476] (rev ba)
15:00.2 SD Host controller [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter [1180:0822] (rev 21)
15:00.4 System peripheral [0880]: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter [1180:0592] (rev 11)
15:00.5 System peripheral [0880]: Ricoh Co Ltd xD-Picture Card Controller [1180:0852] (rev 11)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

The set of devices largely depends on the isolation of the devices, which is represented by the IOMMU groups (http://vfio.blogspot.com/2014/08/iommu- … d-out.html).  So we really need to see what your groups look like (that directory you ran find on before).  In an ideal case, you can just blacklist radeon or use pci-stub for just the GPU.  You may need to apply the ACS patch to break down groups into something usable (at your own risk).  Hybrid laptops like this often have the VGA BIOS for the discrete card buried in the system firmware/ACPI, so you might have fun trying to get it.  Since AMD has a fair success rate as a secondary GPU for the VM, that's the route I'd probably start with to avoid that problem as well as VGA arbitration.  The good news is that your system is old enough that you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The problem is windows 7 requires a CSM present in order to boot, thats why you're stuck on the windows logo, i posted a workarround a few pages back, i recommend you use windows 8 instead.

Does Win8 and Win8.1 have EFI difference?

Now I'm trying to boot Win8.1 ISO, and trying to convert as img file /w FAT32 format and included bootx64.efi.
But both of them can't boot in OVMF, the EFI can't find any bootable devices.

EDIT:

Finally, I started install Windows 8 now.
I use my trump card: Mount USB 2.0 into VM and use USB Stick to install Windows 8, and it worked without any problem.

Last edited by AKSN74 (2015-02-04 04:01:13)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7. And I'm sure that its not caused by HDD.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

Last edited by AKSN74 (2015-02-04 05:09:40)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

PIn vcpus, hugepages, hv-time, etc.

Last edited by nbhs (2015-02-04 05:10:26)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PIn vcpus, hugepages, hv-time, etc.

Thanks for advice!

Now I added hv_time after -cpu host

-cpu host,hv_time,kvm=off

And the result of 3DMARK seems good

3DMARK result Webpage

MrTUX-10

Member

Registered: 2015-01-06

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

PIn vcpus, hugepages, hv-time, etc.

How do you pin the vcpus while using a script instead of virsh? Thanks in advance!

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I got another problem.

After the test, I reboot my VM, gets Win8 crashes (How sucks of Win8......)

So, I re-install Win8 and NVIDIA Driver (347.25)
But now my GTX980 gets "Warning" icon and can't use.

I thought the reason is perhaps the hv_time. Because when I turn it on, it got a same problem at first time. (But solved by re-install driver.)
This time. it's still happening even I re-install driver.

EDIT: Problem solved if delete hv_time and re-install driver.
Really strange, it's not happen before.

Last edited by AKSN74 (2015-02-04 08:12:24)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys congrats to this humongus thread!

I have been wanting to do such a setup for a long time, had some experience with XEN but i bought the wrong cpu for this with no vt-d and i postponed up until now that i am able to buy new hardware and would like to hear what parts to look out for and hopefully have less hassle setting everything up.

The doc with the listed Mainboards https://docs.google.com/spreadsheet/ccc … _web#gid=0 is a huge help but i fear that some info might be missing like info on Revision and the Motheboards i checked so far seem to be a quite a few years old but i am not done with the list yet.

Thanks for any suggestions.

I have good experience with workstation class SuperMicro X9DA7  - no compatibility issues of any kind, can turn on and off ROM of any individual slot, all slots on separate IOMMUs and also builtin hardware (USB and eth), can even redirect BIOS console to serial port. I use it with Xeon E5-2630 v2 and ECC memory.

Lockheed

Member

Registered: 2010-03-16

Posts: 1,522

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

$ find /sys/kernel/iommu_groups
/sys/kernel/iommu_groups
/sys/kernel/iommu_groups/0
/sys/kernel/iommu_groups/0/devices
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1
/sys/kernel/iommu_groups/1/devices
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/2
/sys/kernel/iommu_groups/2/devices
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3
/sys/kernel/iommu_groups/3/devices
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.3
/sys/kernel/iommu_groups/4
/sys/kernel/iommu_groups/4/devices
/sys/kernel/iommu_groups/4/devices/0000:00:1a.0
/sys/kernel/iommu_groups/4/devices/0000:00:1a.1
/sys/kernel/iommu_groups/4/devices/0000:00:1a.2
/sys/kernel/iommu_groups/4/devices/0000:00:1a.7
/sys/kernel/iommu_groups/5
/sys/kernel/iommu_groups/5/devices
/sys/kernel/iommu_groups/5/devices/0000:00:1b.0
/sys/kernel/iommu_groups/6
/sys/kernel/iommu_groups/6/devices
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.1
/sys/kernel/iommu_groups/6/devices/0000:00:1c.2
/sys/kernel/iommu_groups/6/devices/0000:00:1c.3
/sys/kernel/iommu_groups/6/devices/0000:03:00.0
/sys/kernel/iommu_groups/6/devices/0000:04:00.0
/sys/kernel/iommu_groups/6/devices/0000:05:00.0
/sys/kernel/iommu_groups/7
/sys/kernel/iommu_groups/7/devices
/sys/kernel/iommu_groups/7/devices/0000:00:1d.0
/sys/kernel/iommu_groups/7/devices/0000:00:1d.1
/sys/kernel/iommu_groups/7/devices/0000:00:1d.2
/sys/kernel/iommu_groups/7/devices/0000:00:1d.7
/sys/kernel/iommu_groups/8
/sys/kernel/iommu_groups/8/devices
/sys/kernel/iommu_groups/8/devices/0000:00:1e.0
/sys/kernel/iommu_groups/8/devices/0000:15:00.0
/sys/kernel/iommu_groups/8/devices/0000:15:00.2
/sys/kernel/iommu_groups/8/devices/0000:15:00.4
/sys/kernel/iommu_groups/8/devices/0000:15:00.5
/sys/kernel/iommu_groups/9
/sys/kernel/iommu_groups/9/devices
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3

What is better/easier - blacklisting, or using PCI stub? Do I understand it right that you suggest going the pci-stub route because of VGA BIOS being integrated into system firmware?

I plan to use Intel GPU for the host system and give Radeon entirely to qemu. Not sure if that's what you meant by

you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.

?

Also, when I am not running qemu, is there a way to keep using radeon (occasionally) with DRI_PRIME?

EDIT:
I booted with that custom kernel form the OP, but it has negative influence on the radeon GPU temperature. It skyrockets to ~90 C even when not in use and the fan is running like crazy. Perhaps it disables power management for it?

Last edited by Lockheed (2015-02-04 10:41:32)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[  250.730914] kernel BUG at drivers/pci/ats.c:138!

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability.  Maybe the device never really recovers from reset?

Got it. I suppose I should complain to AMD support ...

so I asked Sapphire support (it's their card) but no reply yet. I'm tempted to experiment with a quirk for the device to disable ATS detection on boot. Any hints where to start?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I got another problem.

After the test, I reboot my VM, gets Win8 crashes (How sucks of Win8......)

So, I re-install Win8 and NVIDIA Driver (347.25)
But now my GTX980 gets "Warning" icon and can't use.

I thought the reason is perhaps the hv_time. Because when I turn it on, it got a same problem at first time. (But solved by re-install driver.)
This time. it's still happening even I re-install driver.

EDIT: Problem solved if delete hv_time and re-install driver.
Really strange, it's not happen before.

As reported countless times in this thread, the latest Nvidia driver is incompatible with hyper-v extensions, including hv_time.  The driver doesn't seem to detect these extensions on every boot, which can lead to confusing issues like this, but hv_time and friends should never be recommended for Nvidia users.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

I can confirm that I no longer have DMAR issues with 3.18.5