[SOLVED] XWayland performance regression with kernel 6.10

gromit · 2024-08-01 14:16:30

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc5.r1080.gd65bfb9-1-x86_64.pkg.tar.zst

Nikolai5 · 2024-08-01 16:05:58

gromit wrote:

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc5.r1080.gd65bfb9-1-x86_64.pkg.tar.zst

Kernel: 6.9.0-rc5-1-mainline-01080-gd65bfb9546eb

Seems fine, not been able to replicate issue after 52 minutes uptime.

As an aside, I'm happy to keep going and also appreciate your effort, obviously how many I get through per day will ebb and flow, but what are we talking here? A few more? or like 50 more?

Kateykat · 2024-08-01 17:19:04

Can someone else test if, on a bad linux kernel, rootful XWayland demonstrates the same issue? You can use xwayland-run to create a new rootful xwayland instance:

xwayland-run -decorate -geometry 800x600 -- gimp           # or other software

Nikolai5 · 2024-08-01 18:45:02

Kateykat wrote:

Can someone else test if, on a bad linux kernel, rootful XWayland demonstrates the same issue? You can use xwayland-run to create a new rootful xwayland instance:
xwayland-run -decorate -geometry 800x600 -- gimp           # or other software

I installed the kwin version as I'm on KDE Plasma.
I couldn't get into the file picker as its too big, so increased to 1920x1080 and I could only see the top of the application, rest of it was black space, but I could at least get into file -> open and into the file picker at this resolution and the performance was fine.

This was while the performance was terrible when running normally.

I'm not sure what this tells you but hopefully it progresses the investigation

OpusOne · 2024-08-01 22:03:55

gromit wrote:

Please try the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r5188.g46c6d2b-1-x86_64.pkg.tar.zst

Does this one exclude the commit that I linked to earlier ( a68c7eaa7a8ffdec9287ba1561a668d674c20a13 )?

gromit · 2024-08-02 09:15:52

Nikolai5 wrote:

As an aside, I'm happy to keep going and also appreciate your effort, obviously how many I get through per day will ebb and flow, but what are we talking here? A few more? or like 50 more?

According to git bisect there roughly 9 more steps:

$ git bisect good
Bisecting: 562 revisions left to test after this (roughly 9 steps)

The next step in the bisection is the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc5.r732.gc3c5ac4-1-x86_64.pkg.tar.zst

OpusOne wrote:

Does this one exclude the commit that I linked to earlier ( a68c7eaa7a8ffdec9287ba1561a668d674c20a13 )?

No as I had thought Kateykat reported their own results as bogus in https://bbs.archlinux.org/viewtopic.php … 7#p2187437, but I guess that was just regarding whether NVIDIA-based systems are affected or not I also was not able to revert that commit on top of 6.10 and 6.11-rc1.

But I guess since you already have a finished bisection you could go ahead and open an issue for this on the DRM gitlab issue tracker: https://gitlab.freedesktop.org/drm/amd/-/issues/

Before you do that it would be good if you could test the latest mainline release:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.11rc1-1-x86_64.pkg.tar.zst

Nikolai5 · 2024-08-02 14:29:42

gromit wrote:

The next step in the bisection is the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc5.r732.gc3c5ac4-1-x86_64.pkg.tar.zst

Hi Gromit,

Kernel : 6.9.0-rc5-1-mainline-00732-gc3c5ac4bd7d7

Seems fine / good

Kateykat · 2024-08-02 15:34:54

gromit wrote:

No as I had thought Kateykat reported their own results as bogus in https://bbs.archlinux.org/viewtopic.php … 7#p2187437, but I guess that was just regarding whether NVIDIA-based systems are affected or not

Indeed that was only regarding my laptop, which is Nvidia based. I did the bisection on my AMD desktop, which resulted in the commit I posted earlier.

My apologies for the confusion! I should have been more specific in my reporting.

gromit · 2024-08-02 15:53:07

So Kateykat, whats the status with the bugrerport? Does the mainline release candidate still have the bug?

Nikolai5 · 2024-08-02 16:26:21

gromit wrote:

So Kateykat, whats the status with the bugrerport? Does the mainline release candidate still have the bug?

I tested 6.11.0-rc1-1-mainline a couple of days ago, and it still had the issue, if that's what you mean.

gromit · 2024-08-02 16:31:29

Alright, thats interesting to hear! I was asking because now somebody has to file a bugreport

OpusOne · 2024-08-02 20:06:12

Nikolai5 wrote:

I would just like to add a video example of the issue that I took yesterday, to make sure that everyone is on the same page, as there are several people here and so I want to add some visual context to ensure we're all experiencing the same issue.
https://www.youtube.com/watch?v=lRzhD-dpW-4

Yes, I can confirm this behaves exactly the same in my case.

It usually takes less than 30 min after boot in my case though. Actually, I think that launching some specific apps tends to make it appear sooner. (In my case it seems to be Xilinx ISE - which I doubt many do use here.)

How much VRAM does your GPU have?

Nikolai5 · 2024-08-02 21:20:47

OpusOne wrote:

How much VRAM does your GPU have?

16GB vram

Nikolai5 · 2024-08-02 21:26:36

gromit wrote:

Alright, thats interesting to hear! I was asking because now somebody has to file a bugreport

Fun times… I will do if forced but I feel like beyond being able to replicate the issue the others have better logs and seem to have found the actual patch issue.

So kinda hoping KateyKat or OpusOne don’t mind raising it.

Kateykat · 2024-08-03 04:16:40

Nikolai5 wrote:

So kinda hoping KateyKat or OpusOne don’t mind raising it.

I'll raise it. I'm trying to collect more information / details at the moment, too. My Nvidia laptop has decided to show the same symptoms again (so I'm not insane ) and I'm trying to decipher whether it was my computer being weird (I think nvidia has a similar bug in their proprietary driver I remember seeing when I was initially researching this issue).

I'm also trying to determine if this should be considered a kernel regression or if XWayland needs to update to adapt to newer kernels.

OpusOne · 2024-08-03 04:45:56

Thanks Kateykat, perfect.

Yes, I think NVidia drivers have problems of their own too so that may be a different problem. Or there may still be a link.

And you're right, it may be that the burden is on XWayland instead. I don't know about you guys, but I haven't been able to reproduce the problem in any other configuration than with apps using XWayland. I haven't noticed a regression with GPU benchmarks on their own - but only ran a couple of them. You could try doing that too (glmark2 for instance) and see.

Last edited by OpusOne (2024-08-03 04:46:34)

Nikolai5 · 2024-08-05 11:14:34

OpusOne wrote:

I haven't noticed a regression with GPU benchmarks on their own - but only ran a couple of them. You could try doing that too (glmark2 for instance) and see.

I haven't noticed any performance regression outside of what has already been mentioned. For example I am still able to play video games just fine. GPU performing as expected.

V1del · 2024-08-05 12:24:51

I can't say I can reproduce this... FWIW someone suggested a regression in BTRFS, are you/is the file dialog opening and listing on a BTRFS filesystem with many files? https://bbs.archlinux.org/viewtopic.php?id=297983

Nikolai5 · 2024-08-05 12:26:00

V1del wrote:

are you/is the file dialog opening and listing on a BTRFS filesystem with many files?

No, I use ext4. It isn't just an issue with the file picker, it affects all UI elements. For example in GIMP the tool picker and sub windows are slow. In Prusa Slicer, navigating through the settings is slow.

Edit: Tested on Kernel: 6.11.0-rc2-1-mainline and still an issue.

Note: Can confirm that running in X11 mode there is no issue. So the idea that it's xwayland is sound.
If anyone wants to try to replicate the issue, you have to keep using your computer, opening and closing applications, for instance repeatedly open and close GIMP after running your PC for 20 minutes, and you'll see it degrade.

Last edited by Nikolai5 (2024-08-05 16:40:28)

OpusOne · 2024-08-05 22:30:38

Nothing to do with btrfs which I don't use either. And we have definitely linked it to XWayland.

It's absolutely and definitely an issue between kernel(/amdgpu) changes and XWayland. As discussed before, I was suspecting a particular commit on amdgpu (something about clearing VRAM pages), but got a bit confused in the end with the results of the bisections. And I personally don't know how to build say, the latest kernel (6.10.3) with just this particular commit reverted to its 6.9.x state. Guess at worst, I could grab the latest kernel source code and just manually update the related files, but it's all very time-consuming, and I'm not sure anymore it has anything to do with this commit.

As a reminder, the commit I was refering to is this one: https://gitlab.freedesktop.org/agd5f/li … d674c20a13

Last edited by OpusOne (2024-08-05 22:32:58)

Kateykat · 2024-08-06 00:38:48

I have a flame graph that points to the culprit being a large amount of time spent in marking a page as free in drm_buddy - the vast majority of cpu cycles being spent in list_insert_sorted. The commit mentioned during my bisect is also the one that wires AMDGPU up to the DRM buddy allocator. I suspect it's a result of vram fragmentation caused by XWayland allocating and freeing for every pixmap created via CreatePixmap, which for some x11 clients is called on every frame it seems (GTK in particular seems to be the worst offender here).

I'm inclined to raise the issue with XWayland, maybe there's some optimization that can be done on that side. I don't know if increasing efficiency is possible on the drm buddy side.

OpusOne · 2024-08-06 00:59:06

Thanks for the investigation!

I'd be inclined to consider that the issue is with both, as clearly for the same code in XWayland, there is a performance regression and so there must be a change in the DRM buddy allocator that makes it less efficient than before?
But also, optimizing XWayland on that front looks reasonable.

Edit: we may have a track to the suspect here: https://www.phoronix.com/news/DRM-Buddy … e-Tracking

Last edited by OpusOne (2024-08-06 01:20:12)

Nikolai5 · 2024-08-06 07:30:14

Even if KateyKat logs this bug report, what is the short term solution, as it could be months before it gets resolved.

I assume, rather than sitting on an older version of the kernel, the only thing to do is move to X11.

OpusOne · 2024-08-06 08:06:22

Well, it may not take that long to fix. Hopefully.

Inthe meantime, I don't see many solutions either...

1. Stay on a kernel 6.9.
2. Compile a 6.10 kernel yourself if we find a patch to revert just the above change. Not very convenient.
3. Yes, move to a X11 session.
4. Stop using the affected apps. Not very convenient.
5. Look for maybe development versions of the apps that are affected and which would natively support Wayland. I've seen that the dev version of GIMP (2.99.x), for instance, is supposed to support Wayland natively. I haven't tested it yet, but there's an AUR package: https://aur.archlinux.org/packages/gimp-devel

Maybe someone will add to this list.

xerxes_ · 2024-08-06 10:06:30

...
6. Use linux-lts kernel?

Arch Linux

#26 2024-08-01 14:16:30

Re: [SOLVED] XWayland performance regression with kernel 6.10

#27 2024-08-01 16:05:58

Re: [SOLVED] XWayland performance regression with kernel 6.10

#28 2024-08-01 17:19:04

Re: [SOLVED] XWayland performance regression with kernel 6.10

#29 2024-08-01 18:45:02

Re: [SOLVED] XWayland performance regression with kernel 6.10

#30 2024-08-01 22:03:55

Re: [SOLVED] XWayland performance regression with kernel 6.10

#31 2024-08-02 09:15:52

Re: [SOLVED] XWayland performance regression with kernel 6.10

#32 2024-08-02 14:29:42

Re: [SOLVED] XWayland performance regression with kernel 6.10

#33 2024-08-02 15:34:54

Re: [SOLVED] XWayland performance regression with kernel 6.10

#34 2024-08-02 15:53:07

Re: [SOLVED] XWayland performance regression with kernel 6.10

#35 2024-08-02 16:26:21

Re: [SOLVED] XWayland performance regression with kernel 6.10

#36 2024-08-02 16:31:29

Re: [SOLVED] XWayland performance regression with kernel 6.10

#37 2024-08-02 20:06:12

Re: [SOLVED] XWayland performance regression with kernel 6.10

#38 2024-08-02 21:20:47

Re: [SOLVED] XWayland performance regression with kernel 6.10

#39 2024-08-02 21:26:36

Re: [SOLVED] XWayland performance regression with kernel 6.10

#40 2024-08-03 04:16:40

Re: [SOLVED] XWayland performance regression with kernel 6.10

#41 2024-08-03 04:45:56

Re: [SOLVED] XWayland performance regression with kernel 6.10

#42 2024-08-05 11:14:34

Re: [SOLVED] XWayland performance regression with kernel 6.10

#43 2024-08-05 12:24:51

Re: [SOLVED] XWayland performance regression with kernel 6.10

#44 2024-08-05 12:26:00

Re: [SOLVED] XWayland performance regression with kernel 6.10

#45 2024-08-05 22:30:38

Re: [SOLVED] XWayland performance regression with kernel 6.10

#46 2024-08-06 00:38:48

Re: [SOLVED] XWayland performance regression with kernel 6.10

#47 2024-08-06 00:59:06

Re: [SOLVED] XWayland performance regression with kernel 6.10

#48 2024-08-06 07:30:14

Re: [SOLVED] XWayland performance regression with kernel 6.10

#49 2024-08-06 08:06:22

Re: [SOLVED] XWayland performance regression with kernel 6.10

#50 2024-08-06 10:06:30

Re: [SOLVED] XWayland performance regression with kernel 6.10

Board footer