strategy to narrow down cause of core dumps [SOLVED]

graysky · 2024-01-19 19:25:10

I am experiencing coredump several of which are extreme enough to cause xfce4-terminal to freeze and sddm to reset crash/putting me back to the login screen. Just looking back in

% journalctl --grep="of user 1000" > coredump.txt.

% grep 'of user 1000' coredump.txt 
Jan 18 17:04:43 quadruple systemd-coredump[3658043]: Process 2787211 (xfce4-terminal) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2977]: Process 1360 (polkit-gnome-au) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2982]: Process 1349 (xfce4-power-man) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2981]: Process 1346 (xfce4-notifyd) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2979]: Process 1300 (Thunar) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2976]: Process 1314 (panel-2-cpugrap) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2986]: Process 1102 (xfce4-session) of user 1000 dumped core.
Jan 19 11:33:06 quadruple systemd-coredump[2980]: Process 1351 (solaar) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5864]: Process 1342 (polkit-gnome-au) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5863]: Process 1333 (xfce4-power-man) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5862]: Process 1329 (xfce4-notifyd) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5858]: Process 1282 (Thunar) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5861]: Process 1296 (panel-2-cpugrap) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5866]: Process 1084 (xfce4-session) of user 1000 dumped core.
Jan 19 12:53:17 quadruple systemd-coredump[5859]: Process 1334 (solaar) of user 1000 dumped core.
Jan 19 14:11:03 quadruple systemd-coredump[1827]: Process 1271 (panel-2-cpugrap) of user 1000 dumped core.

Full coredump.txt.

One thought I had is to roll back my packages to the day before these began occurring using the Rollback Machine archive. If I can run stably on a snapshot of packages from say 17-Jan, my thought is that one of the updated packages might be to blame. Is there a more direct strategy to figure out the cause of these?

Last edited by graysky (2024-01-20 16:40:12)

Trilby · 2024-01-19 19:29:04

Most of the stack traces of active threads are ending on a call to g_slice_free_chain_with_offset which is a glib function. Was glib updated in the time frame when this might have started?

You could also try exporting G_DEBUG=gc-friendly before any of your GUI is started (so that all processes inherit this setting) and see if there's any change.

Last edited by Trilby (2024-01-19 19:30:29)

graysky · 2024-01-19 20:17:41

Hi Trilby - from my journalctl, I believe the first occurrence was 18-Jan 17:04. I do not see an update to glib in the month of Jan.

% grep 'upgraded glib' /var/log/pacman.log | tail -n5
[2023-09-29T13:57:09-0400] [ALPM] upgraded glibc (2.38-4 -> 2.38-5)
[2023-10-04T14:39:52-0400] [ALPM] upgraded glibc (2.38-5 -> 2.38-7)
[2023-10-09T16:12:57-0400] [ALPM] upgraded glib2 (2.78.0-2 -> 2.78.0-3)
[2023-10-26T16:29:45-0400] [ALPM] upgraded glib2 (2.78.0-3 -> 2.78.1-1)
[2023-12-07T12:36:06-0500] [ALPM] upgraded glib2 (2.78.1-1 -> 2.78.3-1)

As to your G_DEBUG suggestion, since I am running sddm as my display-manager, would it make sense to modify that service file with a dropin like this? Do you think that will have the processes inheriting that setting?

 # systemctl edit sddm.service

### Editing /etc/systemd/system/sddm.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Service]
Environment="G_DEBUG=gc-friendly"

### Edits below this comment will be discarded

Last edited by graysky (2024-01-19 20:19:22)

seth · 2024-01-19 22:01:55

https://wiki.archlinux.org/title/Core_d … _core_dump
Esp. get https://wiki.archlinux.org/title/Debuginfod in there

This is across various GTK clients all ending in pretty much

                                                  #0  0x00007faf542e33cd g_slice_free_chain_with_offset (libglib-2.0.so.0 + 0x7d3cd)
                                                  #1  0x00007faf543d82d1 n/a (libgobject-2.0.so.0 + 0x132d1)
                                                  #2  0x00007faf5429f1c7 g_datalist_clear (libglib-2.0.so.0 + 0x391c7)
                                                  #3  0x00007faf543e78b4 g_object_unref (libgobject-2.0.so.0 + 0x228b4)
                                                  #4  0x00007faf54e7a4b6 n/a (libgdk-3.so.0 + 0x964b6)
                                                  #5  0x00007faf54e7ab6b n/a (libgdk-3.so.0 + 0x96b6b)
                                                  #6  0x00007faf54e71adc n/a (libgdk-3.so.0 + 0x8dadc)
                                                  #7  0x00007faf54e6d2b8 n/a (libgdk-3.so.0 + 0x892b8)
                                                  #8  0x00007faf54e70268 n/a (libgdk-3.so.0 + 0x8c268)
                                                  #9  0x00007faf54e14fa9 gdk_display_get_event (libgdk-3.so.0 + 0x30fa9)
                                                  #10 0x00007faf54e70428 n/a (libgdk-3.so.0 + 0x8c428)
                                                  #11 0x00007faf542bff69 n/a (libglib-2.0.so.0 + 0x59f69)
                                                  #12 0x00007faf5431e367 n/a (libglib-2.0.so.0 + 0xb8367)

What sounds eerily like the G_SLICE-is-now-malloc situation and with the general impact it's probably caused by some plugin.
What packages where installed/updated in the critical timeframe and do you get the same crashes w/ a fresh user account?

graysky · 2024-01-19 22:25:39

After downgrading to the package set from 17-Jan, I cannot reproduce the coredumps. When I updated, they are back. Affected packages include:

warning: amd-ucode: downgrading from version 20240115.9b6d0b08-1 to version 20231211.f2e52a1c-1
warning: audit: downgrading from version 4.0-1 to version 3.1.2-1
warning: ffmpeg: downgrading from version 2:6.1.1-2 to version 2:6.1.1-1
warning: geeqie: downgrading from version 2.1-4 to version 2.1-3
warning: gimp: downgrading from version 2.10.36-4 to version 2.10.36-3
warning: gnutls: downgrading from version 3.8.3-1 to version 3.8.2-1
warning: gtk-update-icon-cache: downgrading from version 1:4.12.5-1 to version 1:4.12.4-1
warning: gtk3: downgrading from version 1:3.24.40-1 to version 1:3.24.39-2
warning: gvfs: downgrading from version 1.52.2-2 to version 1.52.2-1
warning: gvfs-afc: downgrading from version 1.52.2-2 to version 1.52.2-1
warning: gvfs-gphoto2: downgrading from version 1.52.2-2 to version 1.52.2-1
warning: iana-etc: downgrading from version 20231228-1 to version 20231117-1
warning: imagemagick: downgrading from version 7.1.1.26-2 to version 7.1.1.26-1
warning: imlib2: downgrading from version 1.12.1-2 to version 1.12.1-1
warning: iputils: downgrading from version 20240117-1 to version 20231222-2
warning: libimobiledevice: downgrading from version 1.3.0-10 to version 1.3.0-9
warning: libjxl: downgrading from version 0.9.1-1 to version 0.8.2-2
warning: libpulse: downgrading from version 17.0-3 to version 17.0-2
warning: linux-firmware: downgrading from version 20240115.9b6d0b08-1 to version 20231211.f2e52a1c-1
warning: linux-firmware-whence: downgrading from version 20240115.9b6d0b08-1 to version 20231211.f2e52a1c-1
warning: numactl: downgrading from version 2.0.17-1 to version 2.0.16-1
warning: openmpi: downgrading from version 4.1.6-2 to version 4.1.6-1
warning: openpmix: downgrading from version 4.2.8-1 to version 4.2.6-1
warning: pam: downgrading from version 1.6.0-3 to version 1.5.3-3
warning: polkit: downgrading from version 124-1 to version 123-1
warning: pulseaudio: downgrading from version 17.0-3 to version 17.0-2
warning: shadow: downgrading from version 4.14.3-1 to version 4.14.2-1
warning: sqlite: downgrading from version 3.45.0-1 to version 3.44.2-2
warning: systemd: downgrading from version 255.2-3 to version 255.2-2
warning: systemd-libs: downgrading from version 255.2-3 to version 255.2-2
warning: systemd-sysvcompat: downgrading from version 255.2-3 to version 255.2-2

I have not tested the fresh account. I will do that and report back.

seth · 2024-01-19 22:40:19

The obvious bummer would be

gtk3: downgrading from version 1:3.24.40-1 to version 1:3.24.39-2

graysky · 2024-01-19 23:15:09

I created a new account and left it idle until the screensaver kicked in/power savings eventually kicked it. No coredumps. Logged out of that account and logged into my normal one (xfce4 with plugins). Again, allowed the screensaver/power savings to kick in and replicated them. Will downgrade gtk only and see if that changes it.

graysky · 2024-01-20 12:03:41

Indeed, downgraded gtk3 (1:3.24.40-1 -> 1:3.24.39-2) seems to be prevented the coredumps (13+ hours now). I opened an upstream bug report. Thank you both for the insightful replies.

EDIT: I see heftig updated the package to include a patch with a suspiciously related title. Trying that now.
EDIT2: seems to have fixed the issue!

Last edited by graysky (2024-01-20 16:40:03)

Arch Linux

#1 2024-01-19 19:25:10

strategy to narrow down cause of core dumps [SOLVED]

#2 2024-01-19 19:29:04

Re: strategy to narrow down cause of core dumps [SOLVED]

#3 2024-01-19 20:17:41

Re: strategy to narrow down cause of core dumps [SOLVED]

#4 2024-01-19 22:01:55

Re: strategy to narrow down cause of core dumps [SOLVED]

#5 2024-01-19 22:25:39

Re: strategy to narrow down cause of core dumps [SOLVED]

#6 2024-01-19 22:40:19

Re: strategy to narrow down cause of core dumps [SOLVED]

#7 2024-01-19 23:15:09

Re: strategy to narrow down cause of core dumps [SOLVED]

#8 2024-01-20 12:03:41

Re: strategy to narrow down cause of core dumps [SOLVED]

Board footer