You are not logged in.

#1 2024-07-24 05:56:22

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Linux 6.10 regression: ext4 filesystem

FYI:

Linux 6.10 regression resulting in a crash when using an ext4 filesystem

https://lkml.org/lkml/2024/7/23/402

I do happen to have one ext4 partition with a 16-char long label. I am not sure what kind of "crash" they are expecting. I do not get anything bad like that, but noticed a performance regression when updating to 6.10 - which I haven't pinpointed yet. Probably unrelated.

Last edited by OpusOne (2024-07-24 05:56:59)

Offline

#2 2024-07-24 14:29:35

davevo
Member
Registered: 2024-07-24
Posts: 1

Re: Linux 6.10 regression: ext4 filesystem

I got hit with it after updating on two ext4 partitions on different drives. I was trying to figure out the pattern, and yup, they both have long labels. Although one is only 15 characters. 
Yesterday was not a fun day at all.  The bufferoverflow finally showed up in the journal after I removed the partitions from fstab and manually mounted the drives from another tty.  Anything from within plasma, and it just locked the whole system.
Thanks for letting us know it was reported and is known about.

Last edited by davevo (2024-07-24 14:31:15)

Offline

#3 2024-07-24 21:44:37

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

Since it can lead to a buffer overflow, as far as I get it, the actual effect can be just anything and it may end up not being detected and still wreak havoc - at least I suspect so.
I haven't seen any corresponding entry in the journal yet. This is a partition that is automatically mounted -  didn't try what you did, manually mounting it, to see if it makes a difference.

Offline

#4 2024-07-24 21:58:42

loqs
Member
Registered: 2014-03-06
Posts: 17,877

Re: Linux 6.10 regression: ext4 filesystem

Fix is in 6.10.1.

Offline

#5 2024-07-26 03:46:51

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

Yep, just updated. The performance regression that I noticed with 6.10 appears to be gone too. I can't say for sure that it's related. But that's good news.
Be prepared for 6.10.2 which should be released shortly as well.

Regarding the performance regression, I spoke too soon. Still there. It's hard to pinpoint - it makes working with some apps pretty laggy.

Last edited by OpusOne (2024-07-26 06:41:23)

Offline

#6 2024-07-26 16:27:52

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

OpusOne, as an example if you install GIMP, do you find the interface runs abysmally slow, and the file picker when saving or opening a file is also laggy? Is that an example of what you're talking about?

As that was one example of the performance issue that I was facing among others.

It seems so far that updating to 6.10.1 has resolved the issues.

Last edited by Nikolai5 (2024-07-26 17:18:13)


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#7 2024-07-26 20:55:56

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

I tried, yes Gimp is noticeably more laggy than usual. But the worst is with KiCad that I use a lot. KiCad is pretty graphics-intensive. But even just opening dialogs seems to be much laggier than normal. And still the same with 6.10.1.
This isn't looking good. But I'm not sure yet how to track this down.

Offline

#8 2024-07-26 22:03:14

loqs
Member
Registered: 2014-03-06
Posts: 17,877

Re: Linux 6.10 regression: ext4 filesystem

OpusOne wrote:

This isn't looking good. But I'm not sure yet how to track this down.

You could bisect linux between 6.9 and 6.10 to determine the causal commit. If you need help with the bisection please feel free to ask.

Offline

#9 2024-07-28 21:45:08

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

I also spoke too soon, I've had to downgrade to 6.9.10.

I've seen other topics related to issues with 6.10 kernel, do you think bisect investigation still required? Worth just waiting for 6.10.2?


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#10 2024-07-28 21:58:41

loqs
Member
Registered: 2014-03-06
Posts: 17,877

Re: Linux 6.10 regression: ext4 filesystem

Nikolai5 wrote:

I've seen other topics related to issues with 6.10 kernel, do you think bisect investigation still required? Worth just waiting for 6.10.2?

As user-space is separate from the kernel you can try 6.10.2.arch1 now with the following without it being a partial upgrade (I am assuming you have no out of tree modules or use DKMS):

pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.10.2.arch1-1-x86_64.pkg.tar.zst  https://archive.archlinux.org/packages/l/linux-headers/linux-headers-6.10.2.arch1-1-x86_64.pkg.tar.zst

With respect to bisection,  how easily can you reproduce the issue?

Offline

#11 2024-07-28 22:43:07

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

I use linux-zen and so use DKMS, although currently the only thing I use with that is the package: virtualbox-host-dkms, I assume there is no issue installing the kernel you linked alongside by zen kernel install since they have different package names?

As for reproducing the issue, it's a performance issue, that affects a number of applications, but they do still work, so its not like I'm going to see an error message, its moreso that I can just see that there is a performance hit when using the programs.
An easy one to test is to run GIMP and go into the file picker, on 6.9 it displays my files and folders instantly, on 6.10 it takes time rendering the file picker window and then even more time listing the files to choose from.

So I can certainly check the issue exists on a version of the kernel, but in regard to useful logs I'm not so sure.

Last edited by Nikolai5 (2024-07-28 23:18:52)


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#12 2024-07-29 00:15:23

loqs
Member
Registered: 2014-03-06
Posts: 17,877

Re: Linux 6.10 regression: ext4 filesystem

Nikolai5 wrote:

I use linux-zen and so use DKMS, although currently the only thing I use with that is the package: virtualbox-host-dkms, I assume there is no issue installing the kernel you linked alongside by zen kernel install since they have different package names?

Correct.

Nikolai5 wrote:

So I can certainly check the issue exists on a version of the kernel, but in regard to useful logs I'm not so sure.

From https://bbs.archlinux.org/viewtopic.php … 1#p2186581 with the headers package added as you use DKMS:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r7370.g33e02dc-1-x86_64.pkg.tar.zst https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-headers-v6.9.r7370.g33e02dc-1-x86_64.pkg.tar.zst

Edit:
If that kernel is bad then you can follow along in the linked thread until you get a different result.

Last edited by loqs (2024-07-29 00:16:49)

Offline

#13 2024-07-29 01:21:37

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

Nikolai5 wrote:

As for reproducing the issue, it's a performance issue, that affects a number of applications, but they do still work, so its not like I'm going to see an error message, its moreso that I can just see that there is a performance hit when using the programs.
An easy one to test is to run GIMP and go into the file picker, on 6.9 it displays my files and folders instantly, on 6.10 it takes time rendering the file picker window and then even more time listing the files to choose from.

Yep, exact same here. Glad that I'm not the only one. In my case, the list of apps I use  that have this performance issue:
GIMP, KiCad and yEd.

yEd in particular is awfull laggy, insanely so (but the other 2 are very significantly impacted as well, just not as badly). I did remember that I already ran into this same issue with an older kernel version, but unfortunately I don't remember which. It was several months ago.

I'm not seeing anything particular in the logs when running these apps. So I have no clue either how to determine the issue, other than testing with a different kernel and just observing the behavior.

I benchmarked disk (NVMe SSD) and GPU, to see if that could explain it. But results look normal.

When lags happen in the above apps, it doesn't look like it's taking any significant CPU.

Maybe of importance is that I use Plasma/Wayland. And two monitors.

Offline

#14 2024-07-29 03:02:21

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

Updated to 6.10.2, and the problem is stll there.

Got the same behavior as before: what led me initially to think 6.10.1 solved it (and I'm guessing the same happened to Nikolai5) is that right after rebooting, the issue was apparently gone. But it came back after running the system for a few minutes.
The same happened again when updating to 6.10.2: it appeared solved for a few minutes after rebooting, but then it came back.

Edit: @Nikolai5: Can you confirm on your side that the performance issue is not present with kernel 6.9.10 (and no other difference)? As the issue only seems to appear after a while (and not immediately after rebooting), as you saw as well, you need to keep using those apps until the problem appears. In my experience, it only needs for those apps to be launched (and quit) a couple times, after which the problem should start appearing.

I have investigated this a little further, remembering that the 'strace' command might be useful here.

So here is what I did, for instance:

strace -T -t gimp 2> gimp-strace.log

Just opened the file picker basically - which would take an abnormal amount of time - then closed it, and quit the app.

I saw what the culprit was, thanks to the timestamps. You should get something similar.

poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN|POLLPRI}], 3, -1) = 1 ([{fd=3, revents=POLLIN}]) <1.522216>
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\6\0v'[\304\203\0003\4\0\0\314\1\0\1\0\0\0\08\0e\0$\0\26\0\20\0\1\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32 <0.000020>

The time inside brackets (1.522216 s) is one of the delays that we observe when using the app.
fd=3 corresponds to the following file: "/tmp/.X11-unix/X1" - which as I get it, is basically the way X11 communicates. It's like X11 has abnormal delays in some cases.

So yes, this is related to X11. And will happen only with some apps (not all) that only support X11, with no native Wayland support.
@Nikolai5: can you let me know if you're also on Wayland?

To me that looks like a XWayland issue. The link with the kernel would be tmpfs, or the GPU driver. I have an AMD GPU (so using amdgpu).

Once Nikolai5 can confirm the couple points I asked above, I will probably start a new thread about it, as in the end it clearly looks unrelated to the ext4 bug that was fixed.

Last edited by OpusOne (2024-07-29 06:42:14)

Offline

#15 2024-07-29 09:01:15

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

Hi OpusOne,

So far I've not been able to replicate the issue even after a few hours of using the computer on 6.9.10, not with GIMP or any other application where I noted the problem.
It's not just load time, for example when in the file picker in GIMP or PrusaSlicer, etc it take time to load the list of files, but when as you move your mouse over the items there's a delay between moving your mouse from one thing to the next and it actually moving the highlight, and clicking on a file is a delay between the click and the response.
I agree with your premise of X11 as that would explain some apps vs others, as I am also using Wayland.
Just as you described, the CPU and GPU are not put under load during this.

I'm running KDE Plasma Wayland with an AMD Radeon RX 7800 XT.
Worth noting I am running the linux-zen kernel.

I'm working on this PC at the moment, so can't install kernel to test using strace at the moment.

Are you happy to go off and create a new thread and handle the investigation, please put a link to it here so I can subscribe to it and let me know if you need me to do that strace or do anything else?

They'll probably want a bisect done, so between us both it does seem like 6.9.10 is fine and 6.10.1 is bad.


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#16 2024-07-29 18:34:57

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

I compiled linux-mainline (6.11.0-rc1-1-mainline) via PKGBUILD https://aur.archlinux.org/packages/linux-mainline and the performance was great at first, but then after 30 minutes or so it was same as it was before, so doesn't like it's fixed there either.


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#17 2024-07-29 23:09:06

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

I'm not sure where it's more appropriate to create the new thread. Here or in the "Applications & Desktop Environments" section.
The link with the kernel seems established, but strictly speaking, this looks like a XWayland issue. You can try a kernel 6.10+ and open a Plasma X11 session instead of Wayland, and see if the problem is still there.

Offline

#18 2024-07-30 09:01:59

Nikolai5
Member
From: North West, England, UK
Registered: 2024-01-27
Posts: 171

Re: Linux 6.10 regression: ext4 filesystem

I'd say here, a moderator will move it if it's really inappropriate, but yeah it could go in either.

I'm knee-deep in work on my computer at the moment (I work from home) so I can't install new kernel and test X11 just yet. I think you've got enough ammunition for a post, if you make one, link me to it and I'll put the results of testing X11 on once I've been able to do it. Seems like you do have to run the computer for at least 30 minutes for it to start playing up.


Desktop: Ryzen 7 1800X | AMD 7800XT | KDE Plasma
MacbookPro-2012 | MATE

Offline

#19 2024-07-31 04:22:13

OpusOne
Member
Registered: 2023-05-31
Posts: 125

Re: Linux 6.10 regression: ext4 filesystem

Opened a dedicated thread here: https://bbs.archlinux.org/viewtopic.php?pid=2187248

Offline

Board footer

Powered by FluxBB