[SOLVED] XWayland performance regression with kernel 6.10

OpusOne · 2024-07-31 04:21:24

Two of us have experienced a performance regression using some X11 apps on Wayland since updating to kernel 6.10. The problem still affects 6.10.2 (and apparently 6.11.0-rc1).

The investigation all started in this thread:
https://bbs.archlinux.org/viewtopic.php?id=297991

With some apps which do not have native Wayland support, in particular GIMP (but I've also experienced it with KiCad), some GUI operations have a huge lag that wasn't present with up to kernel 6.9.10. This lag is not associated with any significant CPU or GPU usage.

As mentioned in the other thread, I tracked it down to XWayland using 'strace'. One can see that the XWayland X11 server sometimes does take up to a couple seconds to respond, which is the cause for the GUI lags. That explains that no CPU usage is noticed during these lags, as this is basically a 'poll' syscall that is waiting for the XWayland server.

The only link I could think of at this point with the kernel is either a change in tmpfs (which XWayland uses for communicating), or in amdgpu (both of us affected have an AMD GPU). Or a combination, of course.
One notable point is that this issue doesn't appear right after rebooting the machine, but only after a while using affected apps. Which made me suspect more tmpfs than amdgpu, as one difference from the 'just booted' state and the system having been used for a while is that tmpfs starts empty at every boot and then is populated by processes accessing resources mounted on it, such as /tmp.

If anyone other than us two is running into the same problem, please chime in.

Last edited by OpusOne (2024-09-17 00:40:22)

Kateykat · 2024-07-31 05:25:03

I'm also experiencing this! I was just able to track it down to upgrading from linux 6.9 -> 6.10 (downgrading back to 6.9 fixed it), and a web search led me to this post (after way too long trying trace the issue to the software I was using / XWayland / GTK / mesa / something).

I'm experiencing this particularly in prusa-slicer / orca-slicer, both of which run in XWayland. Forcing a different app to run in XWayland (gnome-subtitles was one I was testing with) also demonstrated the issue. It seems to occur more with a greater number of UI elements. I've only been able to reproduce this in GTK-based GUI applications that run in XWayland so far.

I also have an AMD GPU (RX 6700XT).

I was able to observe very high CPU usage coming from XWayland however. It's only a single thread that goes nuts, so considering the amount of cores I had it was hard to tell with most CPU usage monitoring utilities. I pinned XWayland to a single core (edit: via taskset) to prevent it being passed around and used bottom to observe that core spiking to 100% whenever the UI had been moused-over. The spiking does not occur on linux 6.9.

I have some perf / hotspot recordings of XWayland and prusa-slicer that I can upload if need be. The most significant finding (I think, not very experienced with perf) is an enormous amount of CPU cycles being spent in various AMD GPU calls (called by XWayland).

EDIT:
I was able to verify this occurs on an Nvidia-based system as well.

Last edited by Kateykat (2024-07-31 05:30:17)

OpusOne · 2024-07-31 06:05:42

Thanks for this report. Interesting to see that it occurs with NVidia drivers as well.

Regarding CPU usage, you're right in that it's hard to tell possibly. In my case, on this machine, I have 14 cores/28 threads. I don't think I noticed one particular core spiking though, but with your added information, I may have complementary tests to do. Also, the issues may trigger slightly different behavior depending on the application and how they use X11. You can test with GIMP, for instance, on your side.

gromit · 2024-07-31 06:31:01

Hmm, this is kinda weird type of error

Is there a definite/reproducible way for you to tell that this input lag is happening?

Were you able to narrow the problem down a bit further i.e. by bisecting to the offending change in the Linux Kernel?
If you need help with that I can also provide you some information or prebuild you some kernels to test

Nikolai5 · 2024-07-31 08:09:26

We haven't done a kernel bisect, all we've discussed is on the thread that OpusOne linked in the original post, the only problem with replicating the issue is that you have to wait a sufficient amount of time while using the computer for it to become a problem, maybe 30 - 40 minutes (didn't time it).

But the issue itself can replicated quite easily by going into Prusa Slicer or GIMP and using the file picker, so File -> Open and that window at boot will be instantaneous as you'd expect but after some time (with the application closed, you don't have to be using it), it will take several seconds to render the window and then scrolling through files is very laggy, even trying to close it takes enough time for the system to produce a "this application is not responding, do you want to kill it" dialog box.

(linux and linux-zen both the same)
Downgrading to 6.9.10 is the short term 'solution', 6.10.1 is affected and I'm pretty sure that's one version up, so if a bisect is done I think it's between those two versions.

Can either of you (OpusOne or KateyKat) do the bisect if that's what gromit suggests, as I use my comptuer for work and it's going to be a busy week this week.

Appreciate all your help / effort.

Last edited by Nikolai5 (2024-07-31 08:09:46)

gromit · 2024-07-31 08:35:44

Does the following build show the lagging issue? I already have a few prebuilt kernels between v6.9 and v6.10 so the first steps will probably be rather quick, afterwards I can build & upload new ones in ~15 min for each

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r7370.g33e02dc-1-x86_64.pkg.tar.zst

Note that this installs a new kernel image "linux-mainline" (as we test between 6.9 and 6.10 as the merge base for the stable series), which requires you to teach your bootloader how to boot the other image and select it on boot. You can afterwards verify if you booted the right thing by doing a "uname -r" and check the output

Last edited by gromit (2024-07-31 08:37:07)

OpusOne · 2024-07-31 08:40:50

I tested 6.10.2 as I said and the problem is still there, and there's no reason to believe the issue will be fixed in 6.10.3 as - at least from what I've found online - we seem to be the only bunch reporting this issue so far.

By "bisecting", I think what they mean is, beyond just testing 6.9.10 (which is ok), 6.10, 6.10.1 and 6.10.3 (which are not), compiling a 6.10 kernel but leaving out some commits until we pinpoint the one(s) causing the problem. Unfortunately, that looks ultra time-consuming, especially since 6.10 was a significant update compared to 6.9.

I usually follow kernel changes via the list of changes they publish (which is very summed up and not detailed), is there a place where we can see the detailed commits from 6.9.10 to 6.10?

Nikolai5 · 2024-07-31 09:01:57

Gromit, if the other's can't I may be able to check one or two per day, I use DKMS though only for virtualbox, I assume as long as I don't use that program it won't cause a problem to not install the headers?

gromit · 2024-07-31 09:15:47

You can also install the headers (they are also uploaded next to the package I have referenced above), but it would be best to test with an untainted kernel (meaning that it does not have any external modules like nvidia drivers or vbox stuff loaded).

Nikolai5 · 2024-07-31 09:18:13

Yeah, no problem, Vbox has nothing to do with the issue, so don't need to run it.

OpusOne · 2024-07-31 09:28:19

May try that too later, but I don't have a ton of time. Also "unfortunately" it doesn't seem to happen on my laptop (on which I could do some testing in parallel, and which boots ultra fast too, so that'd be more convenient for testing). The latter has the exact same version of everything, but of course it's not the same hardware, and it runs on integrated Intel GPU.

Nikolai5 · 2024-07-31 16:39:09

@gromit Yes the linux-mainline kernel you linked above does have the issue. Seems to be 20 minutes, I was just using a bit of firefox, discord, opening and closing a few programs. It gets to the point where some programs become unusable, its obviously not just GIMP, that's just an easy test, Prusa Slicer, Steam Client to an extent. It does seem to be related to X11 things in Wayland as Opus suggested, but he investigated that more than I did. I've just been able to replicate the issue.

Kateykat · 2024-07-31 17:43:34

I can test later today, after work. I only have DKMS for KVMFR so I don't mind removing that for a bit. If I have time I'll try a kernel bisect.

gromit · 2024-07-31 22:26:28

Please try the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r3117.gb850dc2-1-x86_64.pkg.tar.zst

Kateykat · 2024-07-31 22:35:45

I'm partway through a bisect right now. Have you determined that b850dc2 is the offending commit, or is your bisect still in progress?

loqs · 2024-07-31 23:02:42

Kateykat wrote:

I'm partway through a bisect right now. Have you determined that b850dc2 is the offending commit, or is your bisect still in progress?

6.9-3117-gb850dc206a57 is the second bisection step assuming the first bisection point between 6.9 and 6.10 6.9-7370-g33e02dc69afb is bad:

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [0c3836482481200ead7b416ca80c68a29cfdaabd] Linux 6.10
git bisect bad 0c3836482481200ead7b416ca80c68a29cfdaabd
# status: waiting for good commit(s), bad commit known
# good: [a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6] Linux 6.9
git bisect good a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
# bad: [33e02dc69afbd8f1b85a51d74d72f139ba4ca623] Merge tag 'sound-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 33e02dc69afbd8f1b85a51d74d72f139ba4ca623

Kateykat · 2024-07-31 23:44:51

Finished my bisection. The resulting first bad commit is a68c7eaa7a8ffdec9287ba1561a668d674c20a13 - "drm/amdgpu: Enable clear page functionality"

No idea why my Nvidia laptop would be affected based on the commit message but haven't dug into it yet.

Log:

git bisect start
# status: waiting for both good and bad commits
# good: [73c962ea6ded55e53a3412771bcafbe4d651f115] Linux 6.9.10
git bisect good 73c962ea6ded55e53a3412771bcafbe4d651f115
# status: waiting for bad commit, 1 good commit known
# bad: [73c962ea6ded55e53a3412771bcafbe4d651f115] Linux 6.9.10
git bisect bad 73c962ea6ded55e53a3412771bcafbe4d651f115
# bad: [0c3836482481200ead7b416ca80c68a29cfdaabd] Linux 6.10
git bisect bad 0c3836482481200ead7b416ca80c68a29cfdaabd
# bad: [33e02dc69afbd8f1b85a51d74d72f139ba4ca623] Merge tag 'sound-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 33e02dc69afbd8f1b85a51d74d72f139ba4ca623
# good: [b850dc206a57ae272c639e31ac202ec0c2f46960] Merge tag 'firewire-updates-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/ieee1394/linux1394
git bisect good b850dc206a57ae272c639e31ac202ec0c2f46960
# good: [46c6d2b186915176be5acc5d4b6f9793eb32a0c7] Merge tag 'asymmetric-keys-next-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd
git bisect good 46c6d2b186915176be5acc5d4b6f9793eb32a0c7
# good: [d65bfb9546eb627e3c578336355c5b81797f2255] gpu: drm: exynos: hdmi: eliminate uses of of_node_put()
git bisect good d65bfb9546eb627e3c578336355c5b81797f2255
# good: [c3c5ac4bd7d7019f2e3ad1720572d53226fe656e] ASoC: Intel: updates for 6.10 - part7
git bisect good c3c5ac4bd7d7019f2e3ad1720572d53226fe656e
# bad: [4a56c0ed5aa0bcbe1f5f7d755fb1fe1ebf48ae9c] Merge tag 'amd-drm-next-6.10-2024-04-26' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect bad 4a56c0ed5aa0bcbe1f5f7d755fb1fe1ebf48ae9c
# bad: [9aa99bb1977aab5f1a23780673f74db99d982632] drm/loongson: fix build after debugfs include change
git bisect bad 9aa99bb1977aab5f1a23780673f74db99d982632
# good: [bbecb57e28e6fd3666e15142728029b084eee6b2] Merge tag 'exynos-drm-next-for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/daeinki/drm-exynos into drm-next
git bisect good bbecb57e28e6fd3666e15142728029b084eee6b2
# bad: [30ea09a182cb37c4921b9d477ed18107befe6d78] drm/bridge: tc358775: fix support for jeida-18 and jeida-24
git bisect bad 30ea09a182cb37c4921b9d477ed18107befe6d78
# bad: [c1696bf8d5f5389c5312aebf9e3ad0267149cdea] drm/tests: Add a test case for drm buddy clear allocation
git bisect bad c1696bf8d5f5389c5312aebf9e3ad0267149cdea
# good: [0546e01d5a0269f02b4aa227f44b30a5a5558792] dt-bindings: panel-simple-dsi: add Khadas TS050 V2 panel
git bisect good 0546e01d5a0269f02b4aa227f44b30a5a5558792
# good: [e0a200ab4b72afd581bd6f82fc1ef510a4fb5478] drm/edid: Parse topology block for all DispID structure v1.x
git bisect good e0a200ab4b72afd581bd6f82fc1ef510a4fb5478
# good: [5c9837374ecf55a1fa3b7622d365a0456960270f] drm/meson: gate px_clk when setting rate
git bisect good 5c9837374ecf55a1fa3b7622d365a0456960270f
# bad: [a68c7eaa7a8ffdec9287ba1561a668d674c20a13] drm/amdgpu: Enable clear page functionality
git bisect bad a68c7eaa7a8ffdec9287ba1561a668d674c20a13
# good: [96950929eb232038022abd961be46d492d7a6f0f] drm/buddy: Implement tracking clear page feature
git bisect good 96950929eb232038022abd961be46d492d7a6f0f
# first bad commit: [a68c7eaa7a8ffdec9287ba1561a668d674c20a13] drm/amdgpu: Enable clear page functionality

loqs · 2024-07-31 23:55:01

@Kateykat is the amdgpu module loaded on the affected system? If so how are the two GPUs on the system configured?

Kateykat · 2024-08-01 00:58:23

It turns out I am a fool, because the laptop doesn't seem to actually be affected. So strike out that Nvidia report. Sorry

OpusOne · 2024-08-01 01:36:13

So, so far looks like a amdgpu regression.

I have taken a look at discussions around this patch on the mailing list. Looks like possible performance hits were already anticipated by some, but somehow it was still decided to keep it for "security" reasons. At least from what I get.
Do you confirm this commit for "drm/amdgpu: Enable clear page functionality" is the culprit?

Kateykat · 2024-08-01 01:56:35

That's when it started, but its hard to revert just that commit since its a small part of larger work.

OpusOne · 2024-08-01 02:22:28

I see. Well, the relevant commit would be this: https://gitlab.freedesktop.org/agd5f/li … d674c20a13

Nikolai5 · 2024-08-01 10:27:55

gromit wrote:

Please try the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r3117.gb850dc2-1-x86_64.pkg.tar.zst

Hi Gromit,

I've been running that kernel (6.9.0-1-mainline-03117-gb850dc206a57) for 1 hour and 35 minutes so far and I would have expected the issue to materialise by now, but it seems perfectly fine.

So I would mark this one as good.

---------------------------

I would just like to add a video example of the issue that I took yesterday, to make sure that everyone is on the same page, as there are several people here and so I want to add some visual context to ensure we're all experiencing the same issue.

https://www.youtube.com/watch?v=lRzhD-dpW-4
In the video I show how smooth and fast Dolphin is and that's how GIMP normally runs, but I show how GIMP is now running after using the computer for around 30 minutes.
Other applications behave similarly, such as PrusaSlicer, Steam to en extent and Firefox to an extent as well, typing has lag (You can type a sentence and remove your hands from the keyboard and the letters are still being printed to screen) in Firefox during the issue and rendering is slowed somewhat.

Note that this video was recorded when running the first kernel Gromit gave me to run, the second one I reference above seems to be fine.

Last edited by Nikolai5 (2024-08-01 10:33:28)

gromit · 2024-08-01 11:25:01

Please try the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r5188.g46c6d2b-1-x86_64.pkg.tar.zst

Nikolai5 · 2024-08-01 13:27:47

gromit wrote:

Please try the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r5188.g46c6d2b-1-x86_64.pkg.tar.zst

This one seems fine:

Kernel: 6.9.0-1-mainline-05188-g46c6d2b18691

Arch Linux

#1 2024-07-31 04:21:24

[SOLVED] XWayland performance regression with kernel 6.10

#2 2024-07-31 05:25:03

Re: [SOLVED] XWayland performance regression with kernel 6.10

#3 2024-07-31 06:05:42

Re: [SOLVED] XWayland performance regression with kernel 6.10

#4 2024-07-31 06:31:01

Re: [SOLVED] XWayland performance regression with kernel 6.10

#5 2024-07-31 08:09:26

Re: [SOLVED] XWayland performance regression with kernel 6.10

#6 2024-07-31 08:35:44

Re: [SOLVED] XWayland performance regression with kernel 6.10

#7 2024-07-31 08:40:50

Re: [SOLVED] XWayland performance regression with kernel 6.10

#8 2024-07-31 09:01:57

Re: [SOLVED] XWayland performance regression with kernel 6.10

#9 2024-07-31 09:15:47

Re: [SOLVED] XWayland performance regression with kernel 6.10

#10 2024-07-31 09:18:13

Re: [SOLVED] XWayland performance regression with kernel 6.10

#11 2024-07-31 09:28:19

Re: [SOLVED] XWayland performance regression with kernel 6.10

#12 2024-07-31 16:39:09

Re: [SOLVED] XWayland performance regression with kernel 6.10

#13 2024-07-31 17:43:34

Re: [SOLVED] XWayland performance regression with kernel 6.10

#14 2024-07-31 22:26:28

Re: [SOLVED] XWayland performance regression with kernel 6.10

#15 2024-07-31 22:35:45

Re: [SOLVED] XWayland performance regression with kernel 6.10

#16 2024-07-31 23:02:42

Re: [SOLVED] XWayland performance regression with kernel 6.10

#17 2024-07-31 23:44:51

Re: [SOLVED] XWayland performance regression with kernel 6.10

#18 2024-07-31 23:55:01

Re: [SOLVED] XWayland performance regression with kernel 6.10

#19 2024-08-01 00:58:23

Re: [SOLVED] XWayland performance regression with kernel 6.10

#20 2024-08-01 01:36:13

Re: [SOLVED] XWayland performance regression with kernel 6.10

#21 2024-08-01 01:56:35

Re: [SOLVED] XWayland performance regression with kernel 6.10

#22 2024-08-01 02:22:28

Re: [SOLVED] XWayland performance regression with kernel 6.10

#23 2024-08-01 10:27:55

Re: [SOLVED] XWayland performance regression with kernel 6.10

#24 2024-08-01 11:25:01

Re: [SOLVED] XWayland performance regression with kernel 6.10

#25 2024-08-01 13:27:47

Re: [SOLVED] XWayland performance regression with kernel 6.10

Board footer