[SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

CarbonChauvinist · 2018-03-06 22:08:37

I have multiple linux kernels installed (mainline, zen, lts, and regular linux). All of a sudden I am unable to boot into my zen kernel, all other kernels seem unaffected. I get a systemd segfault prior to the point where I enter in my password to unlock my LUKS so the core dump does not appear when running coredumpctl (unless that's limited to only coredumps from the current kernel?)

Here is a transcribed copy of the error I receive:

systemd[1]: segfault at 56347c000000 ip 000072f24d0e9d933 sp 00007ffe4da249d0 error 4 in libc.so.6[7f24d0e1booo+1ae000]
systemd[1]: caught <SEGV>, dumped core as pid 119
systemd[1]: freezing execution

What's interesting to me is that package glibc which owns libc.so.6 has not been touched since January according to pacman logs

$ ag glibc /var/log/pacman.log
...
45966:[2017-11-05 01:30] [ALPM] reinstalled glibc (2.26-6)
47269:[2017-11-05 01:33] [ALPM] reinstalled lib32-glibc (2.26-2)
50815:[2017-11-27 17:58] [ALPM] upgraded glibc (2.26-6 -> 2.26-7)
50820:[2017-11-27 17:58] [ALPM] upgraded lib32-glibc (2.26-2 -> 2.26-7)
50885:[2017-11-28 18:03] [ALPM] upgraded glibc (2.26-7 -> 2.26-8)
50923:[2017-11-28 18:03] [ALPM] upgraded lib32-glibc (2.26-7 -> 2.26-8)
54484:[2017-12-27 20:03] [PACMAN] Running '/usr/bin/pacman -S --noconfirm libzip glibc lib32-glibc libatomic_ops zstd libmm-glib libwbclient ghostscript modemmanager smbclient'
54486:[2017-12-27 20:03] [ALPM] upgraded glibc (2.26-8 -> 2.26-9)
54491:[2017-12-27 20:04] [ALPM] upgraded lib32-glibc (2.26-8 -> 2.26-9)
54844:[2018-01-02 21:01] [PACMAN] Running '/usr/bin/pacman -S --noconfirm kig linux-api-headers boost-libs libtorrent-rasterbar liborcus libkolabxml source-highlight libixion libcmis e2fsprogs gcc gcc-libs gdbm glibc lib32-gcc-libs binutils lib32-glibc lib32-libjpeg-turbo kscreenlocker kdecoration kdeplasma-addons kgamma5 khotkeys kde-gtk-config kinfocenter kmenuedit knetattach kscreen kde-cli-tools ksshaskpass ksysguard kuiserver kwallet-pam kwayland-integration kwin kwrited kactivitymanagerd intel-tbb imagemagick drkonqi systemsettings discover user-manager xdg-desktop-portal-kde libkscreen libksysguard libmagick breeze-gtk breeze sddm-kcm bluedevil milou mpd qt5-base noto-fonts-emoji powerdevil plasma-desktop plasma-integration plasma-nm plasma-pa plasma-sdk plasma-vault plasma-wayland-session plasma-workspace plasma-workspace-wallpapers polkit-kde-agent oxygen ncmpcpp linssid libjpeg6-turbo libindi ceph-libs ceph youtube-dl'
54847:[2018-01-02 21:01] [ALPM] upgraded glibc (2.26-9 -> 2.26-10)
54865:[2018-01-02 21:02] [ALPM] upgraded lib32-glibc (2.26-9 -> 2.26-10)
56804:[2018-01-18 17:52] [PACMAN] Running '/usr/bin/pacman -S --noconfirm ca-certificates-mozilla libcups git glib2 glib2-docs glibc lib32-glibc nspr nss doxygen libisoburn libuv'
56807:[2018-01-18 17:52] [PACMAN] Running '/usr/bin/pacman -S --noconfirm ca-certificates-mozilla libcups git glib2 glib2-docs glibc lib32-glibc nspr nss doxygen libisoburn libuv'
56809:[2018-01-18 17:52] [ALPM] upgraded glibc (2.26-10 -> 2.26-11)
56818:[2018-01-18 17:52] [ALPM] upgraded lib32-glibc (2.26-10 -> 2.26-11)

I've tried simply re-installing linux-zen and also re-running mkinitcpio -p linux-zen to no avail.

I use systemd-boot and here's my loader entry:

title Arch Linux Zen (LVM)
linux /vmlinuz-linux-zen
initrd /intel-ucode.img
initrd /acpi_override_3
initrd /initramfs-linux-zen.img
options rw rd.luks.uuid=145628bb-0138-4b8b-bc94-2d041c756539 rd.luks.name=145628bb-0138-4b8b-bc94-2d041c756539=lvm root=/dev/mapper/lvmvg-root ACPI_TABLE_UPGRADE=y acpi_no_auto_ssdt i915.enable_guc_loading=1 i915.enable_guc_submission=1 scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y acpi_backlight=vendor acpi_osi=Linux quiet

Any insight would be appreciated.

Last edited by CarbonChauvinist (2018-03-08 23:13:08)

loqs · 2018-03-06 22:39:37

Which linux-zen release is affected by this issue? If you downgrade linux-zen does that stop the issue occurring?

CarbonChauvinist · 2018-03-06 23:10:59

linux-zen 4.15.7-1 was the affected version, however downgrading to 4.15.6-1 (both linux-zen and linux-zen-headers) using the arch linux archive had no effect; still getting the same systemd segfault on boot.

loqs · 2018-03-06 23:17:42

Checking the pacman.log what else was updated at the same time or after linux-zen 4.15.7-1? Those would be the next packages I would consider downgrading.

CarbonChauvinist · 2018-03-06 23:45:02

Yes, that makes sense. I'm somewhat addicted to 'pacman -Syu' so it looks like I ran it twice in close succession on one day; the first were all systemd related (besides git).

[2018-03-06 08:02] [PACMAN] Running '/usr/bin/pacman -S -y --'
 [2018-03-06 08:02] [PACMAN] synchronizing package lists
 [2018-03-06 08:03] [PACMAN] Running '/usr/bin/pacman -S -- systemd systemd-sysvcompat git libsystemd'
 [2018-03-06 08:03] [ALPM] transaction started
 [2018-03-06 08:03] [ALPM] upgraded libsystemd (237.64-1 -> 238.0-1)
 [2018-03-06 08:03] [ALPM] upgraded systemd (237.64-1 -> 238.0-1)
 [2018-03-06 08:03] [ALPM] upgraded systemd-sysvcompat (237.64-1 -> 238.0-1)
 [2018-03-06 08:03] [ALPM] upgraded git (2.16.2-1 -> 2.16.2-2)

I downgraded all (libsystemd, systemd and systemd-sysvcompat) back down to versions 237.64-1 and was able to boot Zen successfully, even after re-updating linux-zen and linux-zen-headers back up to 4.15.7-1. Thanks!

So now that we know the issue is with systemd what are the next steps I should take in your opinon? (I'm running testing repos btw)

Scimmia · 2018-03-06 23:52:15

Where are those commands coming from? That looks like a classic partial update to me.

CarbonChauvinist · 2018-03-07 00:42:34

@scimmia
Hmmm, I hope it's not a partial. I assume you're referring to the line

[2018-03-06 08:02] [PACMAN] Running '/usr/bin/pacman -S -y --'

I'm using yay now that pacaur is no longer active. Still update the way I always did in the past only thing changed is I moved from 'pacaur -Syu' to 'yay -Syu' I checked back through my pacman.log and afaict that's the same log entry that's shown for all updates I've done using yay.

Scimmia · 2018-03-07 00:48:08

-Sy followed immediately by -S <packages> sure looks like a partial update. Sounds like yay is doing something strange.

CarbonChauvinist · 2018-03-07 01:26:54

Well, that's mildly distressing. I've been doing partial upgrades since my switch to yay then perhaps? (I installed yay dec 14th 2017) Interesting that an issue only arises now?

 $ ag yay /var/log/pacman.log 
53222:[2017-12-14 22:03] [PACMAN] Running 'pacman -U ./yay-2.219-2-x86_64.pkg.tar.xz'
53224:[2017-12-14 22:04] [ALPM] installed yay (2.219-2)

Installing the libsystemd, systemd and systemd-sysvcompat to their latest versions (238.0-1) using pacman proper still presents the segfault on boot with the linux-zen kernel. I suppose if I've been doing partial upgrades since that time, just trying to reinstall those three packages may not undo any damage that was done?

Interestingly, when I used pacman proper to update those three systemd packages, when rebooting I saw a bunch of messages stating something like 'unable to unmount /tmp/oldroot/....' (scrolls by to fast to document). Trying to look up the logs in journalctl on the next boot (using another kernel other than zen) I see this:

 $ journalctl -k -b -1 -r
-- Logs begin at Fri 2017-09-22 21:42:19 EDT, end at Tue 2018-03-06 20:22:14 EST. --
Mar 06 20:09:42 5510 systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Mar 06 20:09:42 5510 systemd-shutdown[1]: Syncing filesystems and block devices.
Mar 06 20:09:41 5510 kernel: wifi0: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-22)
Mar 06 20:09:41 5510 kernel: wifi0: deauthenticating from 58:6d:8f:3d:f6:a2 by local choice (Reason: 3=DEAUTH_LEAVING)
Mar 06 20:06:23 5510 systemd[1]: Detected architecture x86-64.
Mar 06 20:06:23 5510 systemd[1]: systemd 238 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN +PCRE2 default-hierar>
Mar 06 20:06:23 5510 kernel: systemd: 35 output lines suppressed due to ratelimiting

I'll try and adjust the ratelimiting, downgrade the three systemd packages, boot back into zen, upgrade the systemd packages again and then reboot so I can get the actual errors as they may be valid to the issue at hand?

Okay, so what controls the ratelmiting for journalctl? I've edited /etc/systemd/journald.conf to the following (note I uncommented the two ratelimit lines):

 $ cat /etc/systemd/journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
RateLimitIntervalSec=30s
RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

but keep getting the same message

systemd: 35 output lines suppressed due to ratelimiting

Last edited by CarbonChauvinist (2018-03-07 01:51:33)

loqs · 2018-03-07 01:53:09

@Scimmia possibly this https://github.com/Jguer/yay/blob/master/cmd.go#L450 and this https://github.com/Jguer/yay/blob/master/install.go#L44

Last edited by loqs (2018-03-07 01:58:28)

Scimmia · 2018-03-07 03:06:53

So it's refreshing the database, checking what needs a update, then passing those directly to pacman -S? That would definitely qualify as doing something strange. I really don't understand what these wrapper authors are thinking when they do these things.

CarbonChauvinist, if it's a partial update, just a normal pacman -Syu would fix it, there wouldn't be lasting damage. So systemd is segfaulting in the initramfs, first thing I would try is switching back to a standard busybox based initramfs. I assume all of your initramfs' using systemd?

CarbonChauvinist · 2018-03-07 13:22:33

@scimmia Thanks. Yes, all my initramfs' are using systemd. If I'm understanding you correctly I generated a custom busybox based initramfs for testing by doing the following:

1. Created a custom mkinitcpio conf for use, which switched all systemd hooks to busybox hooks

 $ cat /etc/mkinitcpio-busybox.conf     
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
#MODULES=(intel_agp i915)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=(/etc/modprobe.d/psmouse.conf /etc/modprobe.d/itco_wdt.conf)

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.                                              
##   No autodetection is done.                                                                                         
#    HOOKS=(base udev block filesystems)                                                                               
#                                                                                                                      
##   This setup assembles a pata mdadm array with an encrypted root FS.                                                
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.                                             
#    HOOKS=(base udev block mdadm encrypt filesystems)                                                                 
#                                                                                                                      
##   This setup loads an lvm2 volume group on a usb device.                                                            
#    HOOKS=(base udev block lvm2 filesystems)                                                                          
#                                                                                                                      
##   NOTE: If you have /usr on a separate partition, you MUST include the                                              
#    usr, fsck and shutdown hooks.                                                                                     
#HOOKS=(base udev autodetect modconf keyboard block encrypt lvm2p filesystems keyboard fsck)                           
HOOKS=(base udev keyboard autodetect modconf block keymap encrypt lvm2 filesystems fsck)                               
                                                                                                                       
# COMPRESSION                                                                                                          
# Use this to compress the initramfs image. By default, gzip compression                                               
# is used. Use 'cat' to create an uncompressed image.                                                                  
#COMPRESSION="gzip"                                                                                                    
#COMPRESSION="bzip2"                                                                                                   
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

2. Generate custom initramfs for zen making sure to specify kernel version since I'm doing this for a different kernel than what I'm building the initramfs for:

 $ sudo mkinitcpio -c /etc/mkinitcpio-busybox.conf -k 4.15.7-1-zen -g /boot/initramfs-linux-zen-busybox.img

3. Edit loader entry to point to new .img file (also changed rd.luks to cryptdevice e.g.)

 $ cat /boot/loader/entries/arch-zen-lvm.conf
title Arch Linux Zen (LVM)
linux /vmlinuz-linux-zen
initrd /intel-ucode.img
initrd /acpi_override_3
initrd /initramfs-linux-zen-busybox.img  
options rw cryptdevice=UUID=145628bb-0138-4b8b-bc94-2d041c756539:lvm:allow-discards root=/dev/mapper/lvmvg-root ACPI_TABLE_UPGRADE=y acpi_no_auto_ssdt i915.enable_guc_loading=1 i915.enable_guc_submission=1 scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y acpi_backlight=vendor acpi_osi=Linux quiet

This does provide some improvement, in that I'm now able to at least get my LUKS passphrase prompt to appear (previously was segfaulting before even got to the prompt), but once I enter the passphrase I then get the same segfault as before.

Scimmia · 2018-03-07 14:29:50

So it's not just in the initramfs, as soon as the system gets to the real root and executes systemd, it segfaults, too. I think you'll need to talk to the linux-zen maintainer, file a bug report here (heftig is active with zen upstream, as well).

archimboldo · 2018-03-07 14:51:44

This also happens to me with zen kernel 4.15.7-1 and testing repo's enabled. It appears after upgrading systemd to 238 version.
Same kernel works just fine with systemd 237.64-1 and no testing repos. So i guess this topic should be moved to testing section... Regards

CarbonChauvinist · 2018-03-07 15:24:41

Thanks again Scimmia and all for time and insight. Reported bug as best I could: https://bugs.archlinux.org/task/57741

Scimmia · 2018-03-07 15:30:49

Oh, wait a minute, you said in the bug report you were able to boot mainline and lts, but said nothing about the stock kernel in the "linux" package. The OP here said you could boot "regular linux". Clarify?

CarbonChauvinist · 2018-03-07 15:50:13

Sorry, a regrettable oversight on my part. I simply neglected to include the stock linux kernel in the list of kernels I'm able to successfully boot in the bug report. I can in fact successfully boot the stock 4.15.7-1 linux with the 238.0-1 versions of systemd, systemd-sysvcompat and libsystemd. I assume I should append that to the report?

Last edited by CarbonChauvinist (2018-03-07 15:50:33)

Scimmia · 2018-03-07 15:51:00

I'll take care of it.

loqs · 2018-03-07 17:14:17

You could try and bisect systemd and see which commit from systemd is triggering the bad interaction with linux-zen.

CarbonChauvinist · 2018-03-07 19:22:00

loqs wrote:

You could try and bisect systemd and see which commit from systemd is triggering the bad interaction with linux-zen.

I could certainly try, I assume that would be for systemd package? Since there's no 238 systemd-git version in the AUR I'd need to clone the git directly?

$ yay -Ss systemd-git
...
aur/systemd-git 237.r492.gcb77e1228-1 (+3 0.36%) 
    system and service manager
...

I could then 'git bisect bad' to indicate the current version doesn't work- but, I'm not sure the correct version for the 237.64-1 as when I try 'git bisect bad 237.64-1' I get errors that's an incorrect version number. Not sure where I'd need to get that from?

$ mkdir ~/Downloads/gitsystemd && git clone https://github.com/systemd/systemd ~/Downloads/gitsystemd/
$ cd ~/Downloads/gitsystemd/
$ git bisect start
$ git bisect bad
$ git bisect good 237.64-1
fatal: Needed a single revision
Bad rev input: 237.64-1

loqs · 2018-03-07 19:50:21

git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/systemd"
cp -r packages/trunk/ systemd
cd systemd
makepkg -rsi # build and install locally v238 and check it is broken (ensure you have the two signing keys installed)
cd systemd/src/systemd-stable
git checkout v237
cd ../..
makepkg -ersi # build and install locally v237 and check it is not broken
cd systemd/src/systemd-stable
git bisect start
git bisect good v237
git bisect bad 738ab7502afb7663d9aacdd73e79025aa7cd0a9b #the commit id arch is using for v238
Bisecting: 259 revisions left to test after this (roughly 8 steps)
[4cac89bd7c449dad262e4b306b76611b80fb51cc] test: add tests for TemporaryFileSystem=
$ cd ../..
makepkg -ersi #build and install bisected commit and check result
cd systemd/src/systemd-stable
git bisect $RESULT # where result is good or bad
$ cd ../..
makepkg -ersi #build and install bisected commit and check result (repeat last four lines until git bisect has found the commit)

Alad · 2018-03-07 20:19:01

Scimmia wrote:

So it's refreshing the database, checking what needs a update, then passing those directly to pacman -S? That would definitely qualify as doing something strange. I really don't understand what these wrapper authors are thinking when they do these things.

This appears to be some sordid trend. I've added a warning to the wiki page in the vain hope that people take more heed when using random scripts from the internet to update their system.

Last edited by Alad (2018-03-07 20:19:13)

loqs · 2018-03-08 00:02:37

Systemd 238-2 reverts a commit that is incompatible with MuQSS which linux-zen uses.

CarbonChauvinist · 2018-03-08 20:09:32

@loqs, you are awesome. That is all.

Arch Linux

#1 2018-03-06 22:08:37

[SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#2 2018-03-06 22:39:37

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#3 2018-03-06 23:10:59

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#4 2018-03-06 23:17:42

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#5 2018-03-06 23:45:02

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#6 2018-03-06 23:52:15

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#7 2018-03-07 00:42:34

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#8 2018-03-07 00:48:08

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#9 2018-03-07 01:26:54

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#10 2018-03-07 01:53:09

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#11 2018-03-07 03:06:53

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#12 2018-03-07 13:22:33

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#13 2018-03-07 14:29:50

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#14 2018-03-07 14:51:44

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#15 2018-03-07 15:24:41

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#16 2018-03-07 15:30:49

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#17 2018-03-07 15:50:13

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#18 2018-03-07 15:51:00

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#19 2018-03-07 17:14:17

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#20 2018-03-07 19:22:00

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#21 2018-03-07 19:50:21

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#22 2018-03-07 20:19:01

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#23 2018-03-08 00:02:37

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

#24 2018-03-08 20:09:32

Re: [SOLVED] Linux-Zen: systemd segfault error 4 libc.so.6 - Can't boot

Board footer