You are not logged in.

#1 2020-02-19 21:05:19

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

I have a Ryzen 3900X 32GB DDR4 3600Mhz ram, 1080Ti GPU.
OS Is installed on Samsung NVME 850 Pro.
I've found that if I run an emulator on this system, Citra has been my goto. I experience skipping and incredibly poor performance. It cannot even render Citra at native resolution 320x200 I think, without performance degredation.
I thought or feared that somehow the system got rooted, and so I nuked it and reinstalled it from scratch.
But the issue remains.
To start with I've tried running the emulator compiled from git, I've tried it from Retroarch, I've tried the officially provided flatpak, but it's always the same result. Also FS-UAE and WinUAE, through wine, give similarly horrible performance with sound skipping and dropped frames.

I then installed Windows 10 on the system, and all the emulators ran as expected with Citra performing at 100% up to 8x native resolution.
Then I installed POP! OS with Citra, FS-UAE and Retroarch. Again Citra scaled this time up to 9x native resolution at 100% performance. No issues in any other emulators either.

I then went back to Arch, installed it clean again. Tried again, still the same bad result. I installed TKGlitch kernel with Zen 2 optimization and PDS scheduler. And still the same bad performance.
Then I got really curious. And wondered if this was something unique to Arch somehow? But did I do something wrong in installation despite having used arch for 10 years?
So I installed Manjaro.
And lo and behold, the exact same issues on Manjaro, before and after update, as on Arch, same poor performance.
I've also tried mitigations=off on TKGlitch kernel and standard Arch kernel.
So what the hell? How can I possible diagnose this problem? It seems like there's something wrong in the Arch ecosystem.

Last edited by Zaxth (2020-02-21 19:02:21)

Offline

#2 2020-02-19 21:13:52

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Now I have also tried with XFCE4, Gnome and Plasma on Arch, with removing the DE completely from the system between tests, and it doesn't matter one bit. Still bad performance between each.

Offline

#3 2020-02-19 22:14:44

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Turned off Swap. That didn't do anything, performance still bad.

Offline

#4 2020-02-19 22:41:09

jonno2002
Member
Registered: 2016-11-21
Posts: 684

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

what kernel , driver and package versions does pop! os use compared to arch and manjaro ? distro is completely irrelevant here, its the versions of packages/kernels you should be looking at to see which ones a causing the problem.

Last edited by jonno2002 (2020-02-19 22:43:12)

Offline

#5 2020-02-19 23:25:48

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

jonno2002 wrote:

what kernel , driver and package versions does pop! os use compared to arch and manjaro ? distro is completely irrelevant here, its the versions of packages/kernels you should be looking at to see which ones a causing the problem.

I hadn't actually considered that, I simply tried 5.4 and 5.5
But now I'm on 5.3, same as Pop. And it's smooth as butter. But that means there's a severe performance degredation in 5.4 which noone noticed since it's still in 5.5

Offline

#6 2020-02-19 23:29:13

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Can you locate the causal commit by bisecting between 5.3 and 5.4?

Offline

#7 2020-02-19 23:47:19

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

loqs wrote:

Can you locate the causal commit by bisecting between 5.3 and 5.4?

I would suspect that it's to do with hugepage fallback. Committed on september 28th, But it requires a compile to be sure, and then it's anyones guess if I can make nvidia work with that iteration.

Last edited by Zaxth (2020-02-19 23:49:44)

Offline

#8 2020-02-19 23:58:11

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

nvidia-dkms may well work.  It uses a feature based detection system rather than depending on kernel version.
Edit:
You could also try the boot parameter transparent_hugepage=never

https://www.kernel.org/doc/Documentatio … meters.txt

Last edited by loqs (2020-02-20 00:11:24)

Offline

#9 2020-02-20 02:14:19

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

loqs wrote:

nvidia-dkms may well work.  It uses a feature based detection system rather than depending on kernel version.
Edit:
You could also try the boot parameter transparent_hugepage=never

https://www.kernel.org/doc/Documentatio … meters.txt

Then I was not correct. And it's something else. It looked the most likely suspect though.

Last edited by Zaxth (2020-02-20 03:17:29)

Offline

#10 2020-02-20 15:18:30

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

I would think that more people would be interested in fixing this though. With this bug a lot of emulators are simply not useable on Linux.

Offline

#11 2020-02-20 15:41:54

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,425

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

The things is apparently many people do not see this, so you have to find the issue if no one else has this issue. FWIW everythings just dandy here, but I'm on intel if there's a ryzen specific regression I'm never going to see it.

FWIW emulators have the weird property of having bursts of high cpu load during recompilation followed by not much followed by high cpu in rapid succession. (And generally not that much GPU going at all outside of what's needed for upscaling) They are quite an unusual work load in that regard. I'd suggest you check your cpu frequencies/cpu schedulers and whether changing to eg performance makes a difference or whether there was a change in that regard between the kernel versions.

And yes that is indeed a case that "no one else" would notice if they aren't into emulation it might be that "normal" applications work just fine. To give you some perspective I can run almost everything on the default powersave governor of the intel_pstate scheduler without noticing anything off. Switching to somewhat demanding emulation (higan/bsnes on accuracy in this case) had notable stutters/slow downs because the scheduler didn't get that it should keep the clocks high (due to how unusual that work load is), switching to the performance governor to just keep it at the highest clocks and everything was fine, afaik this has been fixed somewhere down the line and I can keep it on powersave but my memory might be foggy

Last edited by V1del (2020-02-20 17:36:00)

Offline

#12 2020-02-20 20:42:03

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

V1del wrote:

The things is apparently many people do not see this, so you have to find the issue if no one else has this issue. FWIW everythings just dandy here, but I'm on intel if there's a ryzen specific regression I'm never going to see it.

FWIW emulators have the weird property of having bursts of high cpu load during recompilation followed by not much followed by high cpu in rapid succession. (And generally not that much GPU going at all outside of what's needed for upscaling) They are quite an unusual work load in that regard. I'd suggest you check your cpu frequencies/cpu schedulers and whether changing to eg performance makes a difference or whether there was a change in that regard between the kernel versions.

And yes that is indeed a case that "no one else" would notice if they aren't into emulation it might be that "normal" applications work just fine. To give you some perspective I can run almost everything on the default powersave governor of the intel_pstate scheduler without noticing anything off. Switching to somewhat demanding emulation (higan/bsnes on accuracy in this case) had notable stutters/slow downs because the scheduler didn't get that it should keep the clocks high (due to how unusual that work load is), switching to the performance governor to just keep it at the highest clocks and everything was fine, afaik this has been fixed somewhere down the line and I can keep it on powersave but my memory might be foggy

Yeah changing the governor to performance is mandatory, it was one of the first things I did, but in testing I reverted to others to make sure. Most of the time I'm permanently on performance.
But yes, my "detective work" continues, it's a lot of work though! It is useful to know that on Intel you don't have issues. So I need to limit to stuff concerning the cpu and not memory.

Last edited by Zaxth (2020-02-20 20:42:51)

Offline

#13 2020-02-20 21:38:47

Ropid
Member
Registered: 2015-03-09
Posts: 1,069

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Can you describe a simple way to test this for other people? What exactly do I have to install and what commands to I have to run to see the problem here?

Offline

#14 2020-02-20 22:02:51

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Ropid wrote:

Can you describe a simple way to test this for other people? What exactly do I have to install and what commands to I have to run to see the problem here?

My baseline so far has been citra, just because it's an emulator that consistently shows this issue in all kernels past 5.3.
Simply install it. Run a rom at the emulators native internal resolution "400x240", notice if your speed stays at 100%, it should on any recent hardware at that resolution - even an i3 2200 can do this about 5xnative, if it doesn't you have the bug.
It can also be experienced in other emulators but this has been my baseline so far. But yes, it's so horrible that with this bug a Ryzen 3900X cannot keep up with an i3 2200.

Last edited by Zaxth (2020-02-20 22:15:04)

Offline

#15 2020-02-20 23:31:42

Ropid
Member
Registered: 2015-03-09
Posts: 1,069

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Is there a way to test this with open stuff? Are there people writing their own games and creating their own ROMs?

EDIT:

I managed to find some random ROM. Just looking at the opening animations in that thing I found, things seem to run fine. It animated smoothly. The graphics were also not super simple, they looked complicated enough. My system here is a Ryzen 2700X.

Last edited by Ropid (2020-02-20 23:45:33)

Offline

#16 2020-02-21 00:06:51

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Ropid wrote:

Is there a way to test this with open stuff? Are there people writing their own games and creating their own ROMs?

Maybe you could use some demoscenes, e.g. https://www.pouet.net/prod.php?which=82158


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#17 2020-02-21 01:59:05

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Banging my head against the wall with this problem tongue
But I've narrowed it down now. The bad commit sits between august 30th and september 24th.

Last edited by Zaxth (2020-02-21 03:39:29)

Offline

#18 2020-02-21 12:53:01

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

The bad commit is between 147162f from 20th september and 6cb2e9e 21st of september. Getting very close to finding it now.

Offline

#19 2020-02-21 13:12:12

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Are you using git bisect or choosing the commits manually?

Offline

#20 2020-02-21 13:21:49

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

loqs wrote:

Are you using git bisect or choosing the commits manually?

Manually.

Offline

#21 2020-02-21 13:31:02

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

You could swap now using

git bisect start
git bisect good 147162f
git bisect bad 6cb2e9e
Bisecting: a merge base must be tested
[fca11622d600228bec405456f41590155b3a3eca] ASoC: sdm845: remove unneeded semicolon

Offline

#22 2020-02-21 13:47:43

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Found it! It's "56c1e83" https://github.com/torvalds/linux/tree/ … 2478394a8d
Now how do we get it fixed, I couldn't begin to tell you whats wrong with it, only that with this commit emulation performance on Ryzen 2 is absolutely tanked. tongue
It looks very harmless
"Pull printk updates from Petr Mladek:

- Fix off-by-one error when calculating messages that might fit into
   kmsg buffer. It causes occasional omitting of the last message.

- Add missing pointer check in %pD format modifier handling.

- Some clean up

* tag 'printk-for-5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/pmladek/printk:
  ABI: Update dev-kmsg documentation to match current kernel behaviour
  printk: Replace strncmp() with str_has_prefix()
  lib/test_printf: Remove obvious comments from %pd and %pD tests
  lib/test_printf: Add test of null/invalid pointer dereference for dentry
  vsprintf: Prevent crash when dereferencing invalid pointers for %pD
  printk: Do not lose last line in kmsg buffer dump"

And nothing in here seems like it would impact performance.
However the commit prior to this, 0d8006d, is absolutely fine.

Last edited by Zaxth (2020-02-21 13:53:28)

Offline

#23 2020-02-21 13:54:47

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

https://github.com/torvalds/linux/commi … 2478394a8d is a merge commit.
Which one of the commits it merged caused the issue?
Edit:
I think your bisection has gone wrong at some point as the merge commit will be after all the commits it contained are added to the branch.
So you found all the commits the merge pulled in were good but the merge itself which should contain nothing more is bad.

Last edited by loqs (2020-02-21 14:01:49)

Offline

#24 2020-02-21 14:20:09

Zaxth
Member
Registered: 2014-03-24
Posts: 34

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

loqs wrote:

https://github.com/torvalds/linux/commi … 2478394a8d is a merge commit.
Which one of the commits it merged caused the issue?
Edit:
I think your bisection has gone wrong at some point as the merge commit will be after all the commits it contained are added to the branch.
So you found all the commits the merge pulled in were good but the merge itself which should contain nothing more is bad.

Yes, I thought the tree was listed in a sequential fashion, clearly it isn't. Entirely my own fault, should've started with git bisect instead. But I'm doing that now, atleast I do have a date to work from.
Bear with me a moment here, because I need to be sure of this, commits later in the tree does inherit changes of commits earlier in the tree right? If this basic assumption is incorrect, I might as well give up finding it.

Last edited by Zaxth (2020-02-21 14:33:53)

Offline

#25 2020-02-21 14:51:32

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: Observed Intermittent skipping, poor performance, ONLY on Kernel 5.4+!

Yes commits inherits from their parents.
In the repo use the gitk command to visualize the git repository select 56c1e8343494f0a315c99964ea1a952478394a8d
It shows it has two parents f97c81dc6ca5996560b3944064f63fc87eb18d00  and ae88de56a1893bdccc7b5af8c12556de649d675e

Offline

Board footer

Powered by FluxBB