Terrible performance regression with Nvidia 390.25 driver

Batou · 2018-02-15 18:26:13

@Ioqs, thanks so much! Building it now.

BTW, I actually downgraded to 4.14/387 about an hour ago and the difference is speed is incredible. No sluggishness, no slowdowns, you can actually play a video inside of a browser and open another page without stuttering. It's shocking how horrible 390 is.

I downgraded all these packages to these versions if anyone's interested:

pacman -U linux-4.14.15-1-x86_64.pkg.tar.xz linux-headers-4.14.15-1-x86_64.pkg.tar.xz acpi_call-1.1.0-96-x86_64.pkg.tar.xz nvidia-387.34-21-x86_64.pkg.tar.xz nvidia-settings-387.34-1-x86_64.pkg.tar.xz nvidia-utils-387.34-5-x86_64.pkg.tar.xz opencl-nvidia-387.34-5-x86_64.pkg.tar.xz libxnvctrl-387.34-1-x86_64.pkg.tar.xz

Tom B · 2018-02-16 12:40:02

kokoko3k wrote:

On another system i run an outdated arch installation, same motherboard, but instead of a 750Ti, it has an e-vga gtx 1060/3G
So, instead of updating the whole system, i decided to make a basic archiso installation (no UEFI), fully updated with just lxde, networkmanager, firefox,chromium and latest nvidia drivers (nouveau blacklisted in kernel command line) and tried it on that system.
Guess what? No problems here too.
Is there a chance that it is a software issue that does not depend "entirely" on drivers?
For you to test it, i made an image of the live usb i made.
It is quite handy too, (it works with archiso snapshots, allows you to save your customizations).
Everything runs with root account (sorry), so to test chromium you have to start it from lxterminal via "chromium --no-sandbox".
Decompress it and use dd to write it to an usb stick, 16GB at least.
When booting it, select the second entry (snapshot)
-> Download Live usb torrent file <-
Magnet link: magnet:?xt=urn:btih:LXIA6L5CPBUYSCH2O4OZPL2S7H5BXHSO

-EDIT-
Whops, i left /etc/vconsole.conf with the italian layout,sorry

Good idea, thanks for this.

You are completely correct, it's not just a driver issue. I could not replicate the issue with your snapshot. Chromium worked as expected without stuttering.

I thought perhaps KDE plasma was the issue so for a completely fair test I installed plasma and sddm on your snapshot. Butter smooth. I mentioned glxgears earlier. On my installation I was getting around 3,000fps. On your snapshot I got 20,700. Same driver version, same kernel version, same display manager, same desktop environment, same desktop resolution.

Could this be a configuration issue or dependency issue? I'm going to try creating a new user on my real installation and see if a completely new user is also affected.

Edit: Adding a brand new user on the same installation I still get poor performance. Whatever it is, it's system wide, not user configuration specific.

Edit again: One configuration difference you specifically mentioned was blacklisting noveau. I've done that, no difference (it wasn't showing up in lsmod anyway so I didn't expect it to have any affect).

Last edited by Tom B (2018-02-16 13:10:11)

1LordAnubis · 2018-02-16 15:59:16

Batou wrote:

@Ioqs, thanks so much! Building it now.
BTW, I actually downgraded to 4.14/387 about an hour ago and the difference is speed is incredible. No sluggishness, no slowdowns, you can actually play a video inside of a browser and open another page without stuttering. It's shocking how horrible 390 is.
I downgraded all these packages to these versions if anyone's interested:
pacman -U linux-4.14.15-1-x86_64.pkg.tar.xz linux-headers-4.14.15-1-x86_64.pkg.tar.xz acpi_call-1.1.0-96-x86_64.pkg.tar.xz nvidia-387.34-21-x86_64.pkg.tar.xz nvidia-settings-387.34-1-x86_64.pkg.tar.xz nvidia-utils-387.34-5-x86_64.pkg.tar.xz opencl-nvidia-387.34-5-x86_64.pkg.tar.xz libxnvctrl-387.34-1-x86_64.pkg.tar.xz 

Aye, and the worst part for me is when you click to open a new tab in chromium and go to mouse over your frequently visited sites.... .the cursor lags like i'm using 100% SWAP and have terrible latency.... meanwhile i'm up to date, amd fx 8350 and 16 gb ram at 25% usage, swap 0%....
This is the worst I've ever seen of the Nvidia driver, it makes my machine feel like it's running with a Pentium 4 processor in terms of desktop responsiveness, and i'm only using the openbox window manager with some basic stuff running

Yeah, hitting the maximize button on a video..... that gets stuck for a couple of seconds....
Chromium responds alot better if you go into settings and disable hardware acceleration, but once again chromium isn't the only thing affected by this, it's just popular and an easy way to reproduce these issues

Also, as has been mentioned, commenting out in xorg.conf, Option "metamodes" "nvidia-auto-select +0+0 { ForceFullCompositionPipeline = On }"
This does help to a degree, but it's still laggy and it reintroduces screen tearing for me, which is a no go....

Thorsten Reinbold · 2018-02-17 00:01:00

On my system, the problem has been fixed with the updated packages

- nvidia 390.25-11
- linux 4.15.3-2

Chromium is running fine without stutter now.

kokoko3k · 2018-02-17 07:05:14

Tom B wrote:

You are completely correct, it's not just a driver issue. I could not replicate the issue with your snapshot. Chromium worked as expected without stuttering.

That's great to hear, at least now there is something to work on.
It is just a matter of bisect/narrowing down to the offending difference between the real installation and the usb one.
I'd try:
- To test with a root account session on the real installation.
- Test the initrd from the usb on the real installation.
- Spot relevant differences and replicate the usb user environment (command 'set') as much as possible
- The same as above, with nvidia-settings
- Non UEFI boot
- spot differences in loaded kernel modules
- using nvidia or nvidia-dkms, (i don't remember what i put on the usb)

...I know, that's quite a lot of work, but chances of success are pretty high

-edit
...Meanwhile, generating and posting on nvidia forums 2 nvidia-bug-reports from the working and not working installations may help driver devs.

Last edited by kokoko3k (2018-02-17 07:40:00)

deadite66 · 2018-02-17 08:28:53

only had flickering on my 2nd monitor, latest updates appear to have fixed it.

update: flickering still there but greatly reduced.

Last edited by deadite66 (2018-02-17 09:04:38)

pauschuu · 2018-02-17 09:20:19

Thorsten Reinbold wrote:

On my system, the problem has been fixed with the updated packages
- nvidia 390.25-11
- linux 4.15.3-2
Chromium is running fine without stutter now.

I just upgraded my system to the exact same versions.
Restarted.

Chromium is still stuttering as hell. Nothing changed sorry.

Tom B · 2018-02-17 13:15:39

Ok I think I've partly found the cause of the issue:

$ __GL_YIELD=USLEEP glxgears
19161 frames in 5.0 seconds = 3832.110 FPS
20391 frames in 5.0 seconds = 4078.159 FPS
20269 frames in 5.0 seconds = 4053.776 FPS

$ __GL_YIELD= glxgears
104434 frames in 5.0 seconds = 20886.725 FPS
106039 frames in 5.0 seconds = 21207.787 FPS
106188 frames in 5.0 seconds = 21237.465 FPS

I had thought I had tested this earlier in the thread, but mistakenly I had it set in two different locations (user config and profile.d). This is still 30% slower than the snapshot and I still get poor performance in chromium, but it has vastly improved my performance in 3d applications. In one game I was testing (And it's a nice test because it has an in-game rendered cutscene) it was getting 14fps with USLEEP and now get 60fps (which seems to be fps limited by some kind of in-game fps cap rather than vsync).

I am still seeing poor performance in Chromium and 30% less FPS in glxgears than in the snapshot. I'll keep digging, but this has solved my 3d application performance issues.

seth · 2018-02-17 13:26:37

yielding allows the kernel to schedule CPU time to other processes, the default yielding behavior of the nvidia driver is sched_yield().

This has two implications:
a) what you see is "expectable" and the performance of glxgears seems CPU limited. You will most likely see this on the 387.xx drivers as well
b) if you want single process/thread maximum performance and don't care about CPU load/temperature, use __GL_YIELD=NOTHING; since most processes are however multithreaded nowadays, unsetting __GL_YIELD (the default) is probably a good general compromise.

Tom B · 2018-02-17 13:46:22

I've had this setting for months and noticed an instant FPS drop in games going from 387.xx to 390.25. Something has changed in the driver or kernel to affect the behaviour.

seth · 2018-02-17 13:51:09

What you should severly test is whether you get a comparable __GL_YIELD depending performance drop (ratio, not absolute) on the 387.xx and the 390.xx drivers (in order the determine whether the variable now makes a -significantly- larger impact or whether the effect was just scaled down, resulting in crossing 60fps in some game)
The theory would be that the yielding strategy didn't change it's systemaic impact but that you now simply dropped from eg. 120FPS/60FPS to 60FPS/30FPS (where the usleep yielding would simply cause a 50% drop in either case, just now it's annoying)

Tom B · 2018-02-17 13:55:39

seth wrote:

What you should severly test is whether you get a comparable __GL_YIELD depending performance drop (ratio, not absolute) on the 387.xx and the 390.xx drivers (in order the determine whether the variable now makes a -significantly- larger impact or whether the effect was just scaled down, resulting in crossing 60fps in some game)
The theory would be that the yielding strategy didn't change it's systemaic impact but that you now simply dropped from eg. 120FPS/60FPS to 60FPS/30FPS (where the usleep yielding would simply cause a 50% drop in either case, just now it's annoying)

That's a very good point. I'll try to do some further testing.

loqs · 2018-02-17 16:21:05

nvidia is also currently incompatible with 4.16 so that is something else for nvidia to fix.
swiotlb_map_sg_attrs is no longer exported, same for swiotlb_map_sg and the flag swiotlb_in_use appears to have been removed some time ago.
So the function nv_dma_maps_swiotlb appears broken and without a simple fix to determine if swiotlb is in use.
Which means nv_requires_dma_remap is also broken which leads to the following block being broken in kernel/nvidia/nv.c 3140

        /*
         * The contents of the pte_array[] depend on whether or not this device
         * requires DMA-remapping. If it does, it should be the phys addresses
         * used by the DMA-remapping paths, otherwise it should be the actual
         * address that the device should use for DMA (which, confusingly, may
         * be different than the CPU physical address, due to a static DMA
         * offset).
         */
        if (will_remap)
        {
            pte_array[i] = at->page_table[i]->phys_addr;
        }
        else
        {
            pte_array[i] = nv_phys_to_dma(nvl->dev,
                at->page_table[i]->phys_addr);
        }

So how about patching the user of page_table well that appears to be in the binary blob nv-kernel.o_binary.

afader · 2018-02-17 20:08:58

Things are a lot better for me now, after I switched from Xorg to wayland.

Roken · 2018-02-17 22:19:05

Here's a weird thing. Latest testing kernel (4.15.4-1) and nvidia 390.25-12 I'm still by and large failing the vsynctester tests. However, benchmarking with unigine-heaven and unigine-valley is showing a noticable improvement, and if there are any micro stutters, I'm not seeing them.

No sign of video tearing, either.

Now, still using compton for vsync rather than driver level, but is there a small chance vsynctester.com has it wrong?

blispx · 2018-02-18 15:14:05

@Roken check by running chromium/chrome with the --enable-gpu-rasterization flag, it will probably work poorly

I suggest returning to the patched driver in version 387, as my colleagues advised earlier

Last edited by blispx (2018-02-18 15:14:24)

Roken · 2018-02-18 19:06:07

blispx wrote:

I suggest returning to the patched driver in version 387, as my colleagues advised earlier

It does work poorly with the higher resolution BG. The thing is, I have no problems in day to day use. It's only with vsynctester.

Other benchmarks (e.g. unigine-valley and unigine-heaven) are just fine, and day to day usage is peachy, hence my comment.

rob-tech · 2018-02-18 22:03:27

I just rebooted after over a week of using Windows 10, and there is still no improvement after applying the latest updates. Chromium is as broken as it has been with horrible stutter and broken vsync. The vsync is so bad that I now just use Windows if I want to see something on Netflix, since the stutter completely ruins the experience.

It's probably something to do with the VRAM allocation workaround since the previous driver had a bug where excessive VRAM was used. Hopefully the next driver is not too far off and they get if fixed.

That having been said for my next HEDT build I'm going with a high end AMD graphics card (whatever succeeds RX Vega 64) as open source drivers are smoother and better usually. My weak Intel HD4000 gets a constant 60 fps in KDE regardless of effect being used, while the Nvidia can't, even on previous drivers.

Tom B · 2018-02-19 15:32:34

Apologies I've got a lot of real work to today so can't debug this issue in detail at the moment, but using `KWIN_TRIPLE_BUFFER` as described by the wiki rather than USLEEP or ForceFullCompositionPipeLine avoids tearing at the expense of slightly higher VRAM usage.

Chromium is still laggy, but since turning off __GL_YIELD=USLEEP it's far less noticeable.

Last edited by Tom B (2018-02-19 15:41:53)

Enverex · 2018-02-19 16:05:30

The issue is actually a bit more serious; Even if I have nothing running (an empty Openbox session, no windows, no bars, etc) I'll see ~4% CPU usage from the Nvidia processes in top, but more importantly nvidia-smi shows a constant 14% GPU load despite the fact the card is entirely idle (vs 0% CPU usage on the older driver).

In short, it's probably all-round better to roll back the driver as it's got some real issues.

seth · 2018-02-19 22:09:12

Does anybody here have the problem and does NOT use the full composition pipeline? (NOTICE that an active multiscreen setup or viewport transformations like scaling the output will implicitly activate it)

Tom B · 2018-02-20 00:49:55

seth wrote:

Does anybody here have the problem and does NOT use the full composition pipeline? (NOTICE that an active multiscreen setup or viewport transformations like scaling the output will implicitly activate it)

Hmm. I have two monitors and don't have it turned on. How can I check whether it is active? I see tearing until I enable it so assume it's not on and I do have the performance issues.

Batou · 2018-02-20 01:24:13

seth wrote:

Does anybody here have the problem and does NOT use the full composition pipeline? (NOTICE that an active multiscreen setup or viewport transformations like scaling the output will implicitly activate it)

You can't even start Xorg if you have Force(Full)CompositionPipeline enabled. I think you can enable it through nvidia-settings but it's not reliable. I don't it makes any difference wrt to performance either way with 390.25.

BTW, 390.25 is so bad and broken, it should be blacklisted on all distros. Why Arch packagers are still pushing this crap downstream onto users is beyond me. It's totally unusable and it makes Arch unusable as a result.

Last edited by Batou (2018-02-20 01:39:30)

cirrus9 · 2018-02-20 01:58:50

seth wrote:

Does anybody here have the problem and does NOT use the full composition pipeline? (NOTICE that an active multiscreen setup or viewport transformations like scaling the output will implicitly activate it)

I did have the problem after the upgrade to linux-4.15.1-2, and nvidia-390.25-4. However, after upgrading again to linux-4.15.2-1, and nvidia-390.25-7 (both from testing at the time), the problem was pretty much cured. I really didn't notice much problem except with chromium. I am currently running linux-4.15.3-2, and nvidia-390.25-11, (both from testing), and I still notice some artifacts with sound in chromium, but whether this is related to the bug, or just a typical chromium nuance, I don't know. Everything else seems to be okay. I'm not seeing any unsual CPU load, or screen tearing in video.
Chromium has always used more resources than firefox on this system, especially when displaying video within the browser, (this is with cinnamon desktop).

Also, I never did notice much of a problem on my Arch install with KDE as the desktop. Everything else is the same except for the desktop, same machine, just on a different drive. With KDE I'm using SDDM, kwin, and compositing is enabled.

I know that my problems occured on update to linux-4.15.1-2, and nvidia-390.25-4, and were mitigated (mostly), after the next kernel/nvidia driver updates. That is ALL I know. I have no idea if this was all caused by nvidia, or if it had multiple causes. I didn't even try the noveau drivers, as I have used them in the past, and found their performance to be subpar anyway.

Guiluge · 2018-02-20 07:49:00

Batou wrote:

seth wrote:
Does anybody here have the problem and does NOT use the full composition pipeline? (NOTICE that an active multiscreen setup or viewport transformations like scaling the output will implicitly activate it)
You can't even start Xorg if you have Force(Full)CompositionPipeline enabled. I think you can enable it through nvidia-settings but it's not reliable. I don't it makes any difference wrt to performance either way with 390.25.
BTW, 390.25 is so bad and broken, it should be blacklisted on all distros. Why Arch packagers are still pushing this crap downstream onto users is beyond me. It's totally unusable and it makes Arch unusable as a result.

Because it's their job, obviously. And they can't test everything, since there are basically Nvidia techs being paid for that purpose.
I agree with you, these drivers are plain bad, but Nvidia is the only one to blame here.
I can manage with these drivers through Opera by disabling hardware accel, but it's a temporary fix.

Arch Linux

#126 2018-02-15 18:26:13

Re: Terrible performance regression with Nvidia 390.25 driver

#127 2018-02-16 12:40:02

Re: Terrible performance regression with Nvidia 390.25 driver

#128 2018-02-16 15:59:16

Re: Terrible performance regression with Nvidia 390.25 driver

#129 2018-02-17 00:01:00

Re: Terrible performance regression with Nvidia 390.25 driver

#130 2018-02-17 07:05:14

Re: Terrible performance regression with Nvidia 390.25 driver

#131 2018-02-17 08:28:53

Re: Terrible performance regression with Nvidia 390.25 driver

#132 2018-02-17 09:20:19

Re: Terrible performance regression with Nvidia 390.25 driver

#133 2018-02-17 13:15:39

Re: Terrible performance regression with Nvidia 390.25 driver

#134 2018-02-17 13:26:37

Re: Terrible performance regression with Nvidia 390.25 driver

#135 2018-02-17 13:46:22

Re: Terrible performance regression with Nvidia 390.25 driver

#136 2018-02-17 13:51:09

Re: Terrible performance regression with Nvidia 390.25 driver

#137 2018-02-17 13:55:39

Re: Terrible performance regression with Nvidia 390.25 driver

#138 2018-02-17 16:21:05

Re: Terrible performance regression with Nvidia 390.25 driver

#139 2018-02-17 20:08:58

Re: Terrible performance regression with Nvidia 390.25 driver

#140 2018-02-17 22:19:05

Re: Terrible performance regression with Nvidia 390.25 driver

#141 2018-02-18 15:14:05

Re: Terrible performance regression with Nvidia 390.25 driver

#142 2018-02-18 19:06:07

Re: Terrible performance regression with Nvidia 390.25 driver

#143 2018-02-18 22:03:27

Re: Terrible performance regression with Nvidia 390.25 driver

#144 2018-02-19 15:32:34

Re: Terrible performance regression with Nvidia 390.25 driver

#145 2018-02-19 16:05:30

Re: Terrible performance regression with Nvidia 390.25 driver

#146 2018-02-19 22:09:12

Re: Terrible performance regression with Nvidia 390.25 driver

#147 2018-02-20 00:49:55

Re: Terrible performance regression with Nvidia 390.25 driver

#148 2018-02-20 01:24:13

Re: Terrible performance regression with Nvidia 390.25 driver

#149 2018-02-20 01:58:50

Re: Terrible performance regression with Nvidia 390.25 driver

#150 2018-02-20 07:49:00

Re: Terrible performance regression with Nvidia 390.25 driver

Board footer