gnome-shell segfault on libwayland-server.so.0.21.0

MickeyMouse · 2023-03-13 13:41:55

Hello

Since a recent update i get regular libwayland-server segfault :

[ 2781.260910] gnome-shell[1194]: segfault at 0 ip 00007f65049ecaba sp 00007ffcddcaaa10 error 4 in libwayland-server.so.0.21.0[7f65049eb000+8000] likely on CPU 4 (core 0, socket 0)

coredumpctl gdb 1194

Current thread is 1 (Thread 0x7f64ff120b40 (LWP 1194))]
(gdb) bt
#0  0x00007f65049ecaba in  () at /usr/lib/libwayland-server.so.0
#1  0x00007f65049f101c in  () at /usr/lib/libwayland-server.so.0
#2  0x00007f65049ef9e2 in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#3  0x00007f6503d6484d in  () at /usr/lib/libmutter-11.so.0
#4  0x00007f6504b3aafb in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#5  0x00007f6504b975d9 in  () at /usr/lib/libglib-2.0.so.0
#6  0x00007f6504b3a0cf in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#7  0x00007f6503cd0a6b in meta_context_run_main_loop () at /usr/lib/libmutter-11.so.0
#8  0x00005611fd00c48d in  ()
#9  0x00007f6503a3c790 in  () at /usr/lib/libc.so.6
#10 0x00007f6503a3c84a in __libc_start_main () at /usr/lib/libc.so.6
#11 0x00005611fd00c715 in  ()

[10799.247034] gnome-shell[7037]: segfault at 0 ip 00007fb91ae9caba sp 00007ffea94bc880 error 4 in libwayland-server.so.0.21.0[7fb91ae9b000+8000] likely on CPU 2 (core 2, socket 0)

coredumpctl gdb 7037

(gdb) bt
#0  0x00007fb91ae9caba in  () at /usr/lib/libwayland-server.so.0
#1  0x00007fb91aea101c in  () at /usr/lib/libwayland-server.so.0
#2  0x00007fb91ae9f9e2 in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#3  0x00007fb91ad6484d in  () at /usr/lib/libmutter-11.so.0
#4  0x00007fb91bad9afb in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#5  0x00007fb91bb365d9 in  () at /usr/lib/libglib-2.0.so.0
#6  0x00007fb91bad90cf in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#7  0x00007fb91acd0a6b in meta_context_run_main_loop () at /usr/lib/libmutter-11.so.0
#8  0x000055a1a237748d in  ()
#9  0x00007fb91aa3c790 in  () at /usr/lib/libc.so.6
#10 0x00007fb91aa3c84a in __libc_start_main () at /usr/lib/libc.so.6
#11 0x000055a1a2377715 in  ()

I have updated the whole system today and i notice that libglib-2.0 has been updated :

[2023-03-13T08:40:20+0100] [ALPM] upgraded glib2 (2.74.6-1 -> 2.76.0-1)

I did not notice any recent update in wayland nor libmutter.

It looks like the bug could be realated to glib2. I will re-install the old package to test.

Does anyone has a similar bug ?

Where is the appropriate location to report such a bug ?

Thank you

Last edited by MickeyMouse (2023-03-13 13:53:13)

Veldora · 2023-03-15 13:33:14

Salutations @MickeyMouse

Unfortunately, I don't have a solution for you, but it seems, that I have the same problem. (According to my comparisions to my and your gdb output)
Here's mine:

(gdb) bt
#0  0x00007faff8bd9aba in  () at /usr/lib/libwayland-server.so.0
#1  0x00007faff8bde01c in  () at /usr/lib/libwayland-server.so.0
#2  0x00007faff8bdc9e2 in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#3  0x00007faff7f6484d in  () at /usr/lib/libmutter-11.so.0
#4  0x00007faff8d3aafb in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#5  0x00007faff8d975d9 in  () at /usr/lib/libglib-2.0.so.0
#6  0x00007faff8d3a0cf in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#7  0x00007faff7ed0a6b in meta_context_run_main_loop () at /usr/lib/libmutter-11.so.0
#8  0x000055ef2f02548d in  ()
#9  0x00007faff7c3c790 in  () at /usr/lib/libc.so.6
#10 0x00007faff7c3c84a in __libc_start_main () at /usr/lib/libc.so.6
#11 0x000055ef2f025715 in  ()

(gdb) bt full
#0  0x00007faff8bd9aba in  () at /usr/lib/libwayland-server.so.0
#1  0x00007faff8bde01c in  () at /usr/lib/libwayland-server.so.0
#2  0x00007faff8bdc9e2 in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#3  0x00007faff7f6484d in  () at /usr/lib/libmutter-11.so.0
#4  0x00007faff8d3aafb in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#5  0x00007faff8d975d9 in  () at /usr/lib/libglib-2.0.so.0
#6  0x00007faff8d3a0cf in g_main_loop_run () at /usr/lib/libglib-2.0.so.0
#7  0x00007faff7ed0a6b in meta_context_run_main_loop () at /usr/lib/libmutter-11.so.0
#8  0x000055ef2f02548d in  ()
#9  0x00007faff7c3c790 in  () at /usr/lib/libc.so.6
#10 0x00007faff7c3c84a in __libc_start_main () at /usr/lib/libc.so.6
#11 0x000055ef2f025715 in  ()

Say, what happens to your machine, when the segfault appears?
Mine literally just stops working.

When taking a look into my journalctl, it stops every binary with a failure and then somehow reaches `target shutdown`
This only happened during the night to my machine so far, when I've locked it and kept all programs open, in case I have to work during the night-hours.
Yesterday, I had the same error and I had to work at 2 a.m.

I locked my screen and went to bed. And at 10 p.m. the segfault appeared. At 2 a.m. I wondered, why all of my programs were closed.

Might you post your running versions ? (maybe we're having the same versions)
Gnome-Version: ?
Mutter-Version: ?
Running Kernel: ?

I'll go first:
Gnome-Version: 43.3-2 (Wayland)
Mutter-Version: 43.3-2
Running Kernel: 6.2.6.arch1-1

Thanks

seth · 2023-03-15 14:00:46

https://bbs.archlinux.org/viewtopic.php … 5#p2090135

Veldora · 2023-03-15 14:05:59

Hi seth
Thank you for answering

I don't think, the coredumps Uhrenvergleich has, are any similar to mine (and maybe MickeyMouse)

Process 2257 (gnome-shell) of user 1035 dumped core.
Stack trace of thread 2257:
#0  0x00007f1b1cae1aba n/a (libwayland-server.so.0 + 0x8aba)
#1  0x00007f1b1cae601c n/a (libwayland-server.so.0 + 0xd01c)
#2  0x00007f1b1cae49e2 wl_event_loop_dispatch (libwayland-server.so.0 + 0xb9e2)
#3  0x00007f1b1d16484d n/a (libmutter-11.so.0 + 0x16484d)
#4  0x00007f1b1debeafb g_main_context_dispatch (libglib-2.0.so.0 + 0x5aafb)
#5  0x00007f1b1df1b5d9 n/a (libglib-2.0.so.0 + 0xb75d9)
#6  0x00007f1b1debe0cf g_main_loop_run (libglib-2.0.so.0 + 0x5a0cf)
#7  0x00007f1b1d0d0a6b meta_context_run_main_loop (libmutter-11.so.0 + 0xd0a6b)
#8  0x0000560e48da448d n/a (gnome-shell + 0x248d)
#9  0x00007f1b1ce3c790 n/a (libc.so.6 + 0x23790)
#10 0x00007f1b1ce3c84a __libc_start_main (libc.so.6 + 0x2384a)
#11 0x0000560e48da4715 n/a (gnome-shell + 0x2715)

It's more libwayland-server.so
Asking in the gitlab of Wayland, results to a closed Issue, since (according to them) it's a gnome issue.

seth · 2023-03-15 14:26:36

glib is very much at the bottom of the stack - if it indeed is the slice/malloc patch, it has the potential to cause completely random segfualts across the board.
See whether downgrading glib stabilizes the behavior and if so, if you can make it crash w/ the environment variable again.

MickeyMouse · 2023-03-16 14:27:45

Hello Veldora

Veldora wrote:

Say, what happens to your machine, when the segfault appears?
Mine literally just stops working.

The whole graphical environment exits and i lost all graphical applications opened. I get a new empty graphical environment.

Veldora wrote:

When taking a look into my journalctl, it stops every binary with a failure and then somehow reaches `target shutdown`
This only happened during the night to my machine so far, when I've locked it and kept all programs open, in case I have to work during the night-hours.
Yesterday, I had the same error and I had to work at 2 a.m.
I locked my screen and went to bed. And at 10 p.m. the segfault appeared. At 2 a.m. I wondered, why all of my programs were closed.
Might you post your running versions ? (maybe we're having the same versions)
Gnome-Version: ?
Mutter-Version: ?
Running Kernel: ?
I'll go first:
Gnome-Version: 43.3-2 (Wayland)
Mutter-Version: 43.3-2
Running Kernel: 6.2.6.arch1-1
Thanks

kernel: 6.2.5-arch1-1
Mutter-Version: 43.3-2
extra/gnome-shell 1:43.3-2
wayland: 1.21.0-2

Since I downgrade glib2 the systems is more stable.

Last edited by MickeyMouse (2023-03-16 14:28:13)

Veldora · 2023-03-16 14:54:14

Hi MickeyMouse

MickeyMouse wrote:

The whole graphical environment exits and i lost all graphical applications opened. I get a new empty graphical environment.

Okay, so we're both having the same issue here.
There's a similar issue described here in this gnome-gitlab issue:
https://gitlab.gnome.org/GNOME/mutter/-/issues/2696

I have recompiled the currently available version of glib2 by editing the PKGBUILD of glib2 with the debug-options enabled.
But I'm unsure, if the debug-mode works, since it didn't happen last night. So I have to wait, until the segfault happens again.

MickeyMouse wrote:

kernel: 6.2.5-arch1-1
Mutter-Version: 43.3-2
extra/gnome-shell 1:43.3-2
wayland: 1.21.0-2

Okay, it seems that we're both having almost the same versions running.

MickeyMouse wrote:

Since I downgrade glib2 the systems is more stable.

Unfortunately, downgrading the version is not possible for me, (and only a temporary solution) since I don't know, if the system becomes more unstable after doing this task.
I will further investigate it and let you know, if I found something helpful together with the gnome-team.

Kind regards
Veldora

seth · 2023-03-16 15:09:13

Unfortunately, downgrading the version is not possible for me, (and only a temporary solution) since I don't know, if the system becomes more unstable after doing this task.

PSA: It will not.

rant incoming…/
The gnome team has really shit the bed this time (again…) by fundamentally altering the memory management of their stupid "malloc wasn't invented here" implementation after enabling and nurturing lazy memory management for almost 2 decades. And by doing that cold. On the unfounded assumption of "iT'S GoNnA Be fInE".
For the next couple of weeks the glib code that is still maintained needs to fix up their memory management, we'll likely find related bugs for the next year(s) and some older (unmaintained) code will simply fall disfunctional.

Whatever the downsides of a downgrade may be - at least temporarily they cannot outfuck this clusterfuck.
And in the long run, you best just move and stay away from anything gnome related as far as possible.
This is batshit crazy.
/rant

Veldora · 2023-03-16 15:17:36

Hi seth

Thank you for your "rant", it really made me smile :)

When you've looked into the GitLab issue, the Arch-notebooks are for the employees.
(So it's not a problem of my personal notebook)

Since they've worked with Ubuntu Gnome until now, it's hard to get them working with, say, KDE Plasma or XFCE.
Because, you know, human beings are creatures of habit.

We could make look KDE Plasma like Gnome, but that's a hell lot of work to do so and maintain without breaking anything in the UI.

I could however try debugging and push the Gnome-Devs to fix this bug. So maybe, it's gonna work, when the debug-version of glib2 is working for me.

Thanks
Veldora

Last edited by Veldora (2023-03-16 15:19:52)

seth · 2023-03-16 15:35:15

This is *not* a bug in glib2!

glib2 in 2005 or so introduced some high level memory management to deal w/ (actual) issues in malloc.
The consequence was that code using this approach to alloc/free memory could get away w/ accessing "stale" memory segments, because they were not immediately released.

When memory is accessed from multiple locations and based on event loop handling, maybe even multiple threads you need to pay some attention to keep your data structures up-to date and aligned. It matters when you free stuff, because that cannot happen before everything else got the message to no longer look there.
This was "eased" w/ g_slice because the memory was (often) not in fact freed immediately, effectively defaulting to Qt's ::deleteLater()

Even though I don't think that was sold as a feature, it was a reality and all sorts of code will have relied on that (because memory management can be hard, poking this way made the code not crash, yay!)

glib2 2.76 has effectively removed the training wheels and pushed you from the front yard onto the Autobahn in one move.
But they've not introduced a bug. They just expose all the broken code they've enabled for decades.

Veldora · 2023-03-16 16:06:12

Hi seth

Now I understand. Thank you for clarification.

Veldora · 2023-03-17 09:56:29

It seems that there is a commit for Gnome 44, that has been merged, that should fix the libwayland segfault error.
https://gitlab.gnome.org/GNOME/mutter/- … 40fa83c0da

Would it be possible for the Arch-maintainers, to look things through an possibly add it as a cherry-pick inside the PKGBUILD of mutter 43, as long as Mutter 44 isn't released?

I've already tried it and it seems to be building quite well (well, not quite, I had to pass --nocheck parameter, since it was executing dbus-run-session xvfb-run which failed.)
But the seem to go flawlessly. When installing the package on my test-machine, it didn't crash by now.

Would be a chance to fix it, instead of waiting for Gnome/Mutter 44 to release.

Last edited by Veldora (2023-03-17 09:56:44)

seth · 2023-03-17 11:27:13

Sicne this fixes https://gitlab.gnome.org/GNOME/mutter/-/issues/2570 - were your crashes associated with S3 cycles?

Veldora · 2023-03-17 11:30:53

Hi seth
If you mean with S3 the suspend mode, then yes, it was.
The crashes only happened, when the laptop was suspended and the lid closed.

The logs that I've posted in the GitLab-Issue have been acknowledged to be the same, as described in the issue.

Veldora · 2023-04-20 08:57:48

Hi seth and MickeyMouse

Quick update:
The code of the commit fixed the issue (for me and our employees).
Temporarily I've cloned the official PKGBUILD of mutter, added a git cherry-pick of the commit, recompiled it and put the package on our internal repo.
At the same time, I've wrote an e-mail to the maintainer of the package (I think it's Jan) and in the next morning, the cherry-pick has been added to the official PKGBUILD of mutter 43.

Now it would be nice to know, if the issue of MickeyMouse still persists or has been resolved.
If it's gone, he could close this issue.

Cheers
Veldora

Arch Linux

#1 2023-03-13 13:41:55

gnome-shell segfault on libwayland-server.so.0.21.0

#2 2023-03-15 13:33:14

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#3 2023-03-15 14:00:46

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#4 2023-03-15 14:05:59

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#5 2023-03-15 14:26:36

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#6 2023-03-16 14:27:45

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#7 2023-03-16 14:54:14

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#8 2023-03-16 15:09:13

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#9 2023-03-16 15:17:36

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#10 2023-03-16 15:35:15

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#11 2023-03-16 16:06:12

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#12 2023-03-17 09:56:29

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#13 2023-03-17 11:27:13

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#14 2023-03-17 11:30:53

Re: gnome-shell segfault on libwayland-server.so.0.21.0

#15 2023-04-20 08:57:48

Re: gnome-shell segfault on libwayland-server.so.0.21.0

Board footer