Terrible performance regression with Nvidia 390.25 driver

rob-tech · 2018-02-06 19:47:01

Hi everyone,

Just wanted to bring to light serious regressions in the nvidia driver and see if anyone else is experiencing them, these include:

- Seriously slow image rendering and stuttering while scrolling in chromium, with the website window freezing for up to 3 seconds when loading a larger image (noticeable on larger wikipedia articles which would scroll smoothly with 387, now every time an image comes into view there is bad stutter).
- Broken vsync support in chromium and firefox making it unsuitable for watching any type of video due to stutter (NEC PA241w monitor running at 60 hz instead of the correct 59.95 as before).

I'm going to bring these up to nvidia on their forums (I'm surprised that this passed QA as other people seem to be complaining of the same thing).

Hardware is:

Dell M4700 mobile workstation
Core i7 3940xm cpu
Nvidia Quadro K2000m
16 gb of ram
Samsung 850 pro ssd

KDE Plasma 5.11.5 environment

There should be no lag on this top of the line machine with such basic tasks. I really regret updating now, and am contemplating downgrading the kernel and nvidia. Let's bring this to nvidia's attention and get them to fix it.

EDIT: Forgot to mention my GUI environment

Last edited by rob-tech (2018-02-06 22:32:30)

loqs · 2018-02-06 20:44:21

If the issue is nvidia 390.25 why not downgrade and patch 387.34 keeping linux 4.15?

seth · 2018-02-06 21:31:26

And test it outside plasma, notably w/o the plasma compositor before...

rob-tech · 2018-02-06 22:09:48

Thank you both for replying, I tried with compositing disabled and the performance issues remain, as does the vsync issue (there is less stuttering, but still much worse than before and the refresh rate is wrong). I'm not familiar with patching the driver and since I have Windows 10, I might fall back on that for scenarios where this bug matters.

I have notified nvidia on their official forums and other people also have the same issue so hopefully they look into it.

In the meantime feel free to post your findings if you are affected and discover some sort of workaround.

EDIT: Another quick followup

It appears that glxgears and other apps excluding chromium and firefox seem to vsync correctly, and nvidia control panel also reports the correct refresh rate. So not the end of the world here. Firefox also seems to work quickly after refreshing the profile, only chromium performance and vsync issues in the web browsers remain.

Kwin is also smooth and tearing free as are downloaded HD videos in vlc. So what changed to break chromium? That is the million dollar question.

Last edited by rob-tech (2018-02-06 22:38:45)

gavinhungry · 2018-02-07 01:08:18

@rob-tech,

I encountered the same issue on my workstation today. If you don't want to downgrade your kernel to 4.14, patch nvidia-387.34 with support for 4.15, and add nvidia and nvidia-utils to IgnorePkg in /etc/pacman.conf.

Modified PKGBUILD: https://gist.github.com/gavinhungry/f02 … b334c01c38
nvidia-utils-387.34: https://archive.archlinux.org/packages/ … pkg.tar.xz

Last edited by gavinhungry (2018-02-07 09:11:51)

rob-tech · 2018-02-07 02:05:12

Thank You, gavinhungry for the advice, I think I'll go with this route as this has been bothering me and I don't like Windows 10 that much.

Just to clarify, I would run makepkg in the same directory as the PKGBUILD file and it would pull in the patch and build a pacman package from nvidia's file. Is this correct or do I need to do some additional steps?

Also what would happen with nvidia's 32 bit compatibility libraries which I upgraded to 390.25, can they remain or would I need to remove them, or patch and downgrade them somehow? The reason I ask this is because I don't want to cause some conflict or unbootable situation.

rob-tech · 2018-02-07 04:39:15

Anyway, I tried to build the old nvidia driver and was really close except the build error-ed out at the end and I don't know why. Please see my screenshots for reference https://imgur.com/a/Romxm, I would really appreciate advice.

I have created the following post at Nvidia forums:

https://devtalk.nvidia.com/default/topi … e-issues-/

As well as other people's posts outlining our issue:

https://devtalk.nvidia.com/default/topi … on-390-25/
https://devtalk.nvidia.com/default/topi … on-390-25/

Please confirm the issue if you have a spare moment as that forum is frequented by nvidia employees.

afader · 2018-02-07 05:21:56

I also have this issue. I am trying to build gavinhungry's downgrade patch now.

gavinhungry · 2018-02-07 07:56:11

@rob-tech,

Your build failed because your build directory has spaces in it (looks to be "Nvidia 387 rollback"). Try renaming it to "nvidia-387-rollback" - I tried building it with your directory name and got the same error.

What 32-bit nvidia libraries do you have installed that you need? What is the output of `pacman -Qqs nvidia`? lib32-nvidia-utils can be gotten from the archive: https://archive.archlinux.org/packages/ … pkg.tar.xz

EDIT:

I'm mirroring the patch on Gist since Curl seems to want to download it as gzip from the original location. This should build you the package in a clean chroot (install the devtools package first):

git clone https://gist.github.com/gavinhungry/f02cb60acc971b6709c462b334c01c38 nvidia-387.34
cd nvidia-387.34

curl -O https://archive.archlinux.org/packages/n/nvidia-utils/nvidia-utils-387.34-5-x86_64.pkg.tar.xz
extra-x86_64-build -c -- -I nvidia-utils-387.34-5-x86_64.pkg.tar.xz

Last edited by gavinhungry (2018-02-07 09:23:20)

deadite66 · 2018-02-07 17:01:13

Have the flickering on my 2nd monitor, noticed it goes away after i come out of suspend.

Pryka · 2018-02-07 17:57:08

I can confirm this. Same issue after upgrade to 390.25 driver. Huge performance drop. Video stuttering, v-sync drop.

I'm on Deepin Environment.

Ps. Hello everyone. My first post on Arch community forum.

Last edited by Pryka (2018-02-07 18:10:22)

rob-tech · 2018-02-07 19:47:52

gavinhungry wrote:

@rob-tech,
Your build failed because your build directory has spaces in it (looks to be "Nvidia 387 rollback"). Try renaming it to "nvidia-387-rollback" - I tried building it with your directory name and got the same error.
What 32-bit nvidia libraries do you have installed that you need? What is the output of `pacman -Qqs nvidia`? lib32-nvidia-utils can be gotten from the archive: https://archive.archlinux.org/packages/ … pkg.tar.xz
EDIT:
I'm mirroring the patch on Gist since Curl seems to want to download it as gzip from the original location. This should build you the package in a clean chroot (install the devtools package first):
git clone https://gist.github.com/gavinhungry/f02cb60acc971b6709c462b334c01c38 nvidia-387.34
cd nvidia-387.34

curl -O https://archive.archlinux.org/packages/n/nvidia-utils/nvidia-utils-387.34-5-x86_64.pkg.tar.xz
extra-x86_64-build -c -- -I nvidia-utils-387.34-5-x86_64.pkg.tar.xz

Thank You gavinhungry for your contribution, I can confirm that this method works and everything is back to normal on the old driver with vsync and performance problems gone. As a precaution I added the following 4 matching nvidia packages to pacman's ignore list:

libxnvctrl
nvidia-utils
nvidia
nvidia-settings

The version numbers are all 387.34 so I think they are meant to be matched. If you need opencl or 32 bit libraries, I think those need to be downgraded to match also. Good luck with fixing your systems (hopefully this method works for you as well). In addition, let's get some activity on nvidia's forum with the threads I mentioned so that they emerge from under their corporate veil and actually do something.

Roken · 2018-02-07 20:35:00

Is this hardware specific? After disabling driver level vsync and letting compton handle vsync, I'm noticing a definite improvement in performance on my GTX680.

rob-tech · 2018-02-08 00:48:03

Hi Roken,

I'm assuming this affects all cards, and your gtx 680 is from the same generation as my k2000m (gt 650m) with more shaders.

If you want to know for sure do as follows:

1) Make sure system load is low, chromium is idle and your compositor's vsync is activated NOT nvidia driver's.
2) Visit https://www.vsynctester.com/
3) Wait approx 10 sec until the animation stabilizes as there will be some judder from the initial web site load.
4) If vsync works correctly the top line will turn green and remain within the +-2 level with only minor fluctuations in this zone, the bottom line will be blue and completely parallel with the top one. The refresh rate at the top will match exactly with nvidia control panel's monitor refresh rate for your hardware (in my case 59.95 but could be 60 or any other value depending on monitor). Finally, looking at the browser window for about 20 seconds the picture panning will be fully smooth with not even the slightest micro stutter as the image pans along.

OR

If vsync fails, the top line will turn red with sharp spikes of latency, there will be persistent stutter that happens every so often and the refresh rate at the top will not match your hardware (see nvidia control panel).

Keep in mind that this is valid for chromium and other chromium based browsers as firefox's vsync implementation is broken and will stutter regardless of your driver version.

Let me know your results and if this helps, also please consider posting in the thread on nvidia forums that I mentioned earlier if it affects you.

Last edited by rob-tech (2018-02-08 00:50:25)

matte3560 · 2018-02-08 01:15:19

I'm also experiencing performance issues with the 390.25 Nvidia driver. Tried both 4.15.1 and the 4.14.17 LTS kernel with stuttering and generally worse performance on both. Can be easily reproduced by opening a constantly updating/changing page in chromium (ex. https://www.gdax.com/trade/BTC-USD) and trying to watch a video in a tab in different browser window. Performance loss and stuttering is also clearly apparent in games, especially when running alongside a busy browser tab.

I have a suspicion that these issues are simply a result of the Spectre mitigations added to the 390 driver (see http://nvidia.custhelp.com/app/answers/detail/a_id/4611). Looking at perf top when running a busy browser tab in chromium with the 390 driver seems to support this theory:

With the old 387 driver under the same system load, nothing in perf top was over 5% and the nv_flush_cache symbol was nowhere in sight. If this the type of performance that can be expected with full Spectre mitigations in place, that is truly devastating. For reference, I'm running a GTX 780 with an Intel i7 3770k CPU.

Last edited by matte3560 (2018-02-08 01:16:39)

rob-tech · 2018-02-08 01:37:47

Surely they can and will do better than this with the Spectre mitigations in place. If this is the type of performance with high end hardware then lower spec PC's will be completely hopeless, in the current state it's almost better to use the cpu in some cases with software acceleration, instead of hardware.

ngoonee · 2018-02-08 02:26:28

rob-tech wrote:

Surely they can and will do better than this with the Spectre mitigations in place. If this is the type of performance with high end hardware then lower spec PC's will be completely hopeless, in the current state it's almost better to use the cpu in some cases with software acceleration, instead of hardware.

That is... totally irrelevent for this topic.

cirrus9 · 2018-02-08 02:45:36

Just wanted to say that I had the same problems. Seemingly not as bad as some reported, but certainly much lower performance than before. I have downloaded, and install the latest kernel, and nvidia modules from testing (kernel - 4.15.2-1) (nvidia - 390.25-7), and the problem seems to be corrected with these versions. Also, the new kernel shows full mitigations for spectre and meltdown.

I'm not telling anyone to experiment with packages from testing, but I couldn't stand the performance problems any longer, and the new versions did fix it for me. I've noticed no other problems so far.

MaddyBoo · 2018-02-08 05:03:12

I was experiencing the same issues - serious graphical lag and screen tearing/flashing, especially on my two external displays, particularly when using Chromium.

Firefox was also very laggy, especially during scrolling, but it wasn't causing tearing/flashing.

I tried https://www.vsynctester.com/ in Chromium, as @rob-tech suggested, and was seeing consistent vsync failures and a framerate of ~15FPS.

I first tried @cirrus9's suggestion of installing testing/linux and testing/nvidia but the issues persisted.

I then followed @gavinhungry's solution which worked perfectly and my system is back to normal.

This was my first time patching a package, so I wasn't exactly sure what to do after running extra-x86_64-build, but eventually figured out I needed to run

pacman -U nvidia-387.34-23-x86_64.pkg.tar.xz nvidia-utils-387.34-5-x86_64.pkg.tar.xz

to install the newly built packages. Then I rebooted and everything worked

Retesting with vsynctester I'm now seeing almost no failures and a framerate of 60FPS.

Thank you all!

Last edited by MaddyBoo (2018-03-03 08:08:40)

kokoko3k · 2018-02-08 08:36:54

That's strange, i'm using 390.25 and do not notice any of those issues.
For the record, i use
* kwin without any tearing prevention
* so no need for __GL_YIELD="USLEEP" or KWIN_TRIPLE_BUFFER=1
* compositing via opengl3.1 backend
* forcefullcompositionpipeline (when i need it)
* i boot my kernel with nopti parameter.
* Dual head setup

System is fully updated (only harfbuzz left behind, but for other reasons)

Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz + Nvidia GTX 750TI

cirrus9 · 2018-02-08 12:40:11

kokoko3k wrote:

That's strange, i'm using 390.25 and do not notice any of those issues.
For the record, i use
* kwin without any tearing prevention
* so no need for __GL_YIELD="USLEEP" or KWIN_TRIPLE_BUFFER=1
* compositing via opengl3.1 backend
* forcefullcompositionpipeline (when i need it)
* i boot my kernel with nopti parameter.
* Dual head setup
System is fully updated (only harfbuzz left behind, but for other reasons)
Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz + Nvidia GTX 750TI

You mention "kwin", so I assume you are using KDE. This got my interest, so I booted up my KDE Arch install, and I don't notice the problems under KDE either, or at least it's not bad enough to be easily noticeable. I have not updated to the testing module versions on the KDE install. My main use Arch install is running Cinnamon. System specs: intel i7-3770, GTX-1050, 32GB RAM. So, these problems may be worse on some desktops/hardware. I also noticed some audio problems under Cinnamon along with the video lags.

Tom B · 2018-02-08 12:42:18

I'm using KDE and am seeing lots of performance issues. Video rendering, mostly but also stuttering in games and browsers. It seems to be when new things are loaded to vram that causes the stutter.

I do have `__GL_YIELD="USLEEP"` set though, so perhaps it's something to do with that. I also haven't yet set nopti. It does seem more like a driver issue to me.

AMD Threadripper 1950x + 980ti, 2x 3840x2160 displays.

Last edited by Tom B (2018-02-08 12:45:33)

cirrus9 · 2018-02-08 12:49:57

Tom B wrote:

I'm using KDE and am seeing lots of performance issues. Video rendering, mostly but also stuttering in games and browsers. It seems to be when new things are loaded to vram that causes the stutter.
I do have `__GL_YIELD="USLEEP"` set though, so perhaps it's something to do with that. I also haven't yet set nopti. It does seem more like a driver issue to me.

AMD Threadripper 1950x + 980ti, 2x 3840x2160 displays.

You mention "nopti", I may be wrong, but I thought "pti" had been enabled way before these problems emerged.

Tom B · 2018-02-08 13:13:06

I had thought it had been introduced in 4.15 which is why it took so long to get released, however you're right, it was introduced a month ago so we can rule that out.

kokoko3k · 2018-02-08 14:54:20

AFAIK, page table isolation has been introduced in 4.14, so nopti has been effective since then.
__GL_YIELD="USLEEP" has proven to be a performance killer in several games in the past (borderlands 2 dropped from 120fps to 20 or 30), so i just removed it and relied to forcefullcompositionpipeline for vsync.

One thing that may be worth trying is experimenting various value for InitialPixmapPlacement.
(nvidia-settings -a InitialPixmapPlacement=$value)

Arch Linux

#1 2018-02-06 19:47:01

Terrible performance regression with Nvidia 390.25 driver

#2 2018-02-06 20:44:21

Re: Terrible performance regression with Nvidia 390.25 driver

#3 2018-02-06 21:31:26

Re: Terrible performance regression with Nvidia 390.25 driver

#4 2018-02-06 22:09:48

Re: Terrible performance regression with Nvidia 390.25 driver

#5 2018-02-07 01:08:18

Re: Terrible performance regression with Nvidia 390.25 driver

#6 2018-02-07 02:05:12

Re: Terrible performance regression with Nvidia 390.25 driver

#7 2018-02-07 04:39:15

Re: Terrible performance regression with Nvidia 390.25 driver

#8 2018-02-07 05:21:56

Re: Terrible performance regression with Nvidia 390.25 driver

#9 2018-02-07 07:56:11

Re: Terrible performance regression with Nvidia 390.25 driver

#10 2018-02-07 17:01:13

Re: Terrible performance regression with Nvidia 390.25 driver

#11 2018-02-07 17:57:08

Re: Terrible performance regression with Nvidia 390.25 driver

#12 2018-02-07 19:47:52

Re: Terrible performance regression with Nvidia 390.25 driver

#13 2018-02-07 20:35:00

Re: Terrible performance regression with Nvidia 390.25 driver

#14 2018-02-08 00:48:03

Re: Terrible performance regression with Nvidia 390.25 driver

#15 2018-02-08 01:15:19

Re: Terrible performance regression with Nvidia 390.25 driver

#16 2018-02-08 01:37:47

Re: Terrible performance regression with Nvidia 390.25 driver

#17 2018-02-08 02:26:28

Re: Terrible performance regression with Nvidia 390.25 driver

#18 2018-02-08 02:45:36

Re: Terrible performance regression with Nvidia 390.25 driver

#19 2018-02-08 05:03:12

Re: Terrible performance regression with Nvidia 390.25 driver

#20 2018-02-08 08:36:54

Re: Terrible performance regression with Nvidia 390.25 driver

#21 2018-02-08 12:40:11

Re: Terrible performance regression with Nvidia 390.25 driver

#22 2018-02-08 12:42:18

Re: Terrible performance regression with Nvidia 390.25 driver

#23 2018-02-08 12:49:57

Re: Terrible performance regression with Nvidia 390.25 driver

#24 2018-02-08 13:13:06

Re: Terrible performance regression with Nvidia 390.25 driver

#25 2018-02-08 14:54:20

Re: Terrible performance regression with Nvidia 390.25 driver

Board footer