[Solved] glib2 (possibly?) related segfault

banana_pancakes · 2017-05-29 10:04:59

Hi,

I'm getting segfaults since yesterday when trying to open certain applications, and I'm unsure on how to debug/fix it. I would appreciate some pointers if you folks have any.

So, here's what happened:

There was an update to VLC yesterday, and pacman failed with:

warning: could not get file information for usr/lib/vlc/plugins/plugins.dat"
[...]
(4/4) Updating the vlc plugin cache...
error: command terminated by signal 11: Segmentation fault

The journal logs show:

kernel: vlc-cache-gen[32715]: segfault at 8 ip 00007f5cdef37503 sp 00007ffeb4491110 error 4 in ld-2.25.so[7f5cdef27000+23000]

followed by a core dump. Note that ld-2.25.so is provided by glibc (according to pacman -Fs ld-2.25.so). I tried reinstalling the glibc and glib2 packages to no avail.

Removing and re-installing VLC gives the same result. I tried to recompile vlc with the vlc-cache-gen hook commented out from the PKGBUILD, but that fails for some other unclear reason during the compilation process.

make[7]: *** [Makefile:2233: libqt4_plugin_la-main_interface.lo] Error 1

After some searching online, I found some threads that seem related from ~3 years ago, such as:

https://bugs.archlinux.org/task/39676
https://bbs.archlinux.org/viewtopic.php?pid=1395040

But trying the suggested "fix"/workaround, which is running the line

sudo LD_PRELOAD=/usr/lib/libgobject-2.0.so.0 pacman -S vlc

did not actually work in this case (same result).

After a while, I noticed that mpd and mpv also crash on startup with a similar segfault in ld-2.25.so (I tried reinstalling them as well, but they still crash on startup)
I also tried the linux-lts kernel just in case, but that was also fruitless.

Some other apps that depend on glib2 (like hexchat and wireshark) seem to work fine, so I'm really not really sure what the problem here is...

The only unusual thing I've done lately that I can think of, is run

sudo btrfs balance start -dusage=70 -v /

the night before I noticed the problem. Had to run this because of disk space issues (had almost no unallocated space left, despite 100gb of unused space).

Does anyone have any clue on how to proceed with debugging this problem?

Last edited by banana_pancakes (2017-06-01 14:32:18)

seth · 2017-05-29 15:57:43

This post lacks all the interesting bits, first and foremost the coredump/backtrace fro the vlc crash.
As of the make error - yes, make exited with 1, but you dropped all the nice lines before that would have told us "why" ;-)

This line however "warning: could not get file information for usr/lib/vlc/plugins/plugins.dat"" is ultra-fishy.
=>

stat /usr/lib/vlc/plugins/plugins.dat
pacman -Qkk vlc

And in doubt, you're entire pacman log, please (use some pastebin service, it's quite some text)
https://wiki.archlinux.org/index.php/Li … in_clients

banana_pancakes · 2017-05-29 22:38:59

Hi, thanks for the reply! - So, the stacktrace isn't terribly interesting, which is why I originally omitted it:

systemd-coredump[22013]: Process 22011 (vlc) of user 1000 dumped core.
Stack trace of thread 22011:
#0 0x00007f583710547d n/a (n/a)

Same for mpv and mpd.

However, the backtrace (via gdb) has a little bit more information:

(gdb) bt
#0 0x00007f042a1b947d in match_symbol () at /lib64/ld-linux-x86-64.so.2
#1 0x00007f042a1b9980 in _dl_check_map_versions () at /lib64/ld-linux-x86-64.so.2
#2 0x00007f042a1bc4d3 in dl_open_worker () at /lib64/ld-linux-x86-64.so.2
#3 0x00007f04298e7c94 in _dl_catch_error () at /usr/lib/libc.so.6
#4 0x00007f042a1bbf29 in _dl_open () at /lib64/ld-linux-x86-64.so.2
#5 0x00007f0429b66f26 in () at /usr/lib/libdl.so.2
#6 0x00007f04298e7c94 in _dl_catch_error () at /usr/lib/libc.so.6
#7 0x00007f0429b675f5 in () at /usr/lib/libdl.so.2
#8 0x00007f0429b66fb1 in dlopen () at /usr/lib/libdl.so.2
#9 0x00007f0429567bc0 in () at /usr/lib/libvlccore.so.8
#10 0x00007f042954bf5b in () at /usr/lib/libvlccore.so.8
#11 0x00007f042954c288 in () at /usr/lib/libvlccore.so.8
#12 0x00007f042954c1e4 in () at /usr/lib/libvlccore.so.8
#13 0x00007f042954c764 in () at /usr/lib/libvlccore.so.8
#14 0x00007f042954cd6d in () at /usr/lib/libvlccore.so.8
#15 0x00007f04294d2f81 in libvlc_InternalInit () at /usr/lib/libvlccore.so.8
#16 0x00007f0429f8fcee in libvlc_new () at /usr/lib/libvlc.so.5
#17 0x0000000000401256 in ()
#18 0x00007f04297e143a in __libc_start_main () at /usr/lib/libc.so.6
#19 0x00000000004014ba in ()

The stacktrace for vlc-cache-gen is the same as that for vlc. The one for vlc-cache-gen is generated when attempting to install vlc (which runs vlc-cache-gen as a hook). The vlc coredump is generated while trying to run vlc.

For mpd:

(gdb) bt
#0 0x00007f4c54e3847d in match_symbol () from /lib64/ld-linux-x86-64.so.2
#1 0x00007f4c54e38980 in _dl_check_map_versions () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f4c54e38cf0 in _dl_check_all_versions () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f4c54e2ca88 in version_check_doit () from /lib64/ld-linux-x86-64.so.2
#4 0x00007f4c54e403db in _dl_receive_error () from /lib64/ld-linux-x86-64.so.2
#5 0x00007f4c54e2b8bb in dl_main () from /lib64/ld-linux-x86-64.so.2
#6 0x00007f4c54e3f388 in _dl_sysdep_start () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f4c54e2cd38 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f4c54e28d88 in _start () from /lib64/ld-linux-x86-64.so.2
#9 0x0000000000000001 in ?? ()
#10 0x00007fffd1cee3c4 in ?? ()
#11 0x0000000000000000 in ?? ()

The mpd stacktrace is exactly similar to the mpv one (except memory addresses of course)

[Edit]: The compilation failure was originally due to https://bugs.archlinux.org/task/54222. However, after fixing it, compilation still fails on both vlc or vlc3 (vlc-git from AUR) with:

GEN ../modules/plugins.dat
/bin/sh: line 4: 6070 Segmentation fault (core dumped) ./vlc-cache-gen ../modules
make[2]: *** [Makefile:1527: ../modules/plugins.dat] Error 139
make[2]: Leaving directory '/tmp/makepkg/vlc/src/vlc-2.2.6/bin'
make[1]: *** [Makefile:2265: all-recursive] Error 1
make[1]: Leaving directory '/tmp/makepkg/vlc/src/vlc-2.2.6'
make: *** [Makefile:2150: all] Error 2
==> ERROR: A failure occurred in build().
Aborting...

The pacman logs when trying to install vlc are simply:

:: Proceed with installation? [Y/n]
(1/1) checking keys in keyring [------------------------------------] 100%
(1/1) checking package integrity [------------------------------------] 100%
(1/1) loading package files [------------------------------------] 100%
(1/1) checking for file conflicts [------------------------------------] 100%
(1/1) checking available disk space [------------------------------------] 100%
warning: could not get file information for usr/lib/vlc/plugins/plugins.dat
:: Processing package changes...
(1/1) reinstalling vlc [------------------------------------] 100%
:: Running post-transaction hooks...
(1/4) Updating icon theme caches...
(2/4) Arming ConditionNeedsUpdate...
(3/4) Updating the desktop file MIME type cache...
(4/4) Updating the vlc plugin cache...
error: command terminated by signal 11: Segmentation fault

And in journalctl, one can see the stacktrace that's on the top of this reply.

Note that the file /usr/lib/vlc/plugins/plugins.dat doesn't exist after this failed installation attempt, so the suggested stat command failed.

$ pacman -Qkk vlc
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (No such file or directory)
vlc: 1004 total files, 1 altered file

Last edited by banana_pancakes (2017-05-30 11:18:15)

Ropid · 2017-05-29 23:08:17

You could run btrfs scrub to see if it thinks file contents are corrupted:

sudo btrfs scrub start /

You can track progress like this:

sudo watch btrfs scrub status /

seth · 2017-05-30 06:48:11

There's some bad plugin file, maybe cruft, which vlc attempts to load.
run

strace /usr/lib/vlc/vlc-cache-gen -f /usr/lib/vlc/plugins 2>&1 | grep open

possibly as root (do not sudo strace, you'll just get errors) and see which file is tried to be opened when dlopen segfaults.

banana_pancakes · 2017-05-30 10:54:51

I ran "btrfs scrub", but no errors were ultimately found. Thanks for the suggestion anyway!

I updated my previous reply with the new compilation error (it now fails on running vlc-cache-gen). It's the same one as https://bbs.archlinux.org/viewtopic.php?id=186280, but the links there don't really help.

I've also tried stracing earlier, it doesn't seem to segfault on an open() syscall, but on an mmap. Here's the last few lines of running strace on mpv:

open("/usr/lib/samba/libflag-mapping-samba4.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\10\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=6032, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3d5d10f000
mmap(NULL, 2101264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3d340cf000
mprotect(0x7f3d340d0000, 2093056, PROT_NONE) = 0
mmap(0x7f3d342cf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x7f3d342cf000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3d5d10d000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3d5d10b000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV (core dumped) +++
zsh: segmentation fault (core dumped) strace mpv

and on mpd:

open("/usr/lib/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=469288, ...}) = 0
mmap(NULL, 2564584, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f34bbf30000
mprotect(0x7f34bbfa2000, 2093056, PROT_NONE) = 0
mmap(0x7f34bc1a1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x71000) = 0x7f34bc1a1000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f34dd720000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV (core dumped) +++
zsh: segmentation fault (core dumped) strace mpd

on /usr/lib/vlc/vlc-cache-gen -f /usr/lib/vlc/plugins:

open("/usr/lib/libgmp.so.10", O_RDONLY|O_CLOEXEC) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\5\1\0\0\0\0\0"..., 832) = 832
fstat(5, {st_mode=S_IFREG|0755, st_size=603040, ...}) = 0
mmap(NULL, 2698208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f5f23023000
mprotect(0x7f5f230b5000, 2093056, PROT_NONE) = 0
mmap(0x7f5f232b4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x91000) = 0x7f5f232b4000
close(5) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV (core dumped) +++
zsh: segmentation fault (core dumped) strace /usr/lib/vlc/vlc-cache-gen -f /usr/lib/vlc/plugins

grepped on "open"

open("/usr/lib/samba/libnettle.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnettle.so.6", O_RDONLY|O_CLOEXEC) = 5
open("/usr/lib/samba/libhogweed.so.4", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libhogweed.so.4", O_RDONLY|O_CLOEXEC) = 5
open("/usr/lib/samba/libgmp.so.10", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libgmp.so.10", O_RDONLY|O_CLOEXEC) = 5

on vlc, as non-root, because vlc won't run as root:

open("/usr/lib/libgmp.so.10", O_RDONLY|O_CLOEXEC) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\5\1\0\0\0\0\0"..., 832) = 832
fstat(5, {st_mode=S_IFREG|0755, st_size=603040, ...}) = 0
mmap(NULL, 2698208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7fed467f1000
mprotect(0x7fed46883000, 2093056, PROT_NONE) = 0
mmap(0x7fed46a82000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x91000) = 0x7fed46a82000
close(5) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV (core dumped) +++
zsh: segmentation fault (core dumped) strace vlc

I also tried reinstalling the gmp package, no effect. (grasping at straws)

banana_pancakes · 2017-06-01 14:31:47

I "solved" it by checking all the dependencies of mpd (pacman -Qi mpd, "Depends on") and reinstalling all of them:

sudo pacman -S --asdeps libao  ffmpeg  libmodplug  audiofile  libshout  libmad  curl  faad2  sqlite  jack  libmms wavpack  avahi  libid3tag  yajl  libmpdclient  icu  libupnp  libnfs  libsamplerate  libsoxr smbclient  libcdio-paranoia  libgme

.
After that, all of mpd, mpv and vlc started working.

I'm not sure what caused the problem, but I'm glad it's in the past. The set of common dependencies of all three programs, is, surprisingly, empty. The intersection of mpd and mpv is {'ffmpeg', 'jack', 'libcdio-paranoia'}. However, I think I did try reinstalling ffmpeg earlier, and it didn't help. So I'm ultimately not sure which library caused the problem.

Arch Linux

#1 2017-05-29 10:04:59

[Solved] glib2 (possibly?) related segfault

#2 2017-05-29 15:57:43

Re: [Solved] glib2 (possibly?) related segfault

#3 2017-05-29 22:38:59

Re: [Solved] glib2 (possibly?) related segfault

#4 2017-05-29 23:08:17

Re: [Solved] glib2 (possibly?) related segfault

#5 2017-05-30 06:48:11

Re: [Solved] glib2 (possibly?) related segfault

#6 2017-05-30 10:54:51

Re: [Solved] glib2 (possibly?) related segfault

#7 2017-06-01 14:31:47

Re: [Solved] glib2 (possibly?) related segfault

Board footer