[SOLVED] glibc update bricks system

OdinEidolon · 2017-07-18 12:06:39

After updating glibc to 2.25-7, my system does not start anymore. The kernel boots, but then I am greeted with a kernel panic coming from ACPI or systemd stuff, sometimes the one sometimes the other. This happens with any of the 3 different kernel versions I tried.

The fix is to login from a live usb and copy /lib64/ld-2.25*.so and /usr/lib/libdl-2.25*.so from the live-usb version to the installed OS. Without this, even chrooting would not work as bash would not start (segmentation fault).
I whought it was a corrupted file, but I re-updated and the same happened again. Does anybody else having this problem? Should I report this as a bug?

Last edited by OdinEidolon (2017-07-18 14:30:47)

R00KIE · 2017-07-18 12:23:24

Do you happen to be doing partial updates? Also make sure the mirror you are using is not out of sync.

OdinEidolon · 2017-07-18 12:26:11

R00KIE wrote:

Do you happen to be doing partial updates? Also make sure the mirror you are using is not out of sync.

No it was a full update and this was 2 days ago, so by now it should have synced... still I do not see any update that could be related. My mirrors are generated by reflector weekly so they should be updated.

seth · 2017-07-18 12:50:25

Actual crash message?
Did you forget to mount the /boot partition on an update and are now running an older kernel?

uname -a
pacman -Qs linux
lscpu # let's just see whether thats sth. exotic

If it's an intel CPU, do you have intel-ucode installed and enabled?

OdinEidolon · 2017-07-18 13:04:40

seth wrote:

Actual crash message?
Did you forget to mount the /boot partition on an update and are now running an older kernel?
uname -a
pacman -Qs linux
lscpu # let's just see whether thats sth. exotic
If it's an intel CPU, do you have intel-ucode installed and enabled?

uname -a
Linux XPS15af 4.11.11-1-ck-skylake #1 SMP PREEMPT Sat Jul 15 13:14:40 EDT 2017 x86_64 GNU/Linux

pacman -Qs linux
local/acpi_call-dkms 1.1.0-7
    A linux kernel module that enables calls to ACPI methods through /proc/acpi/call
local/alsa-lib 1.1.4.1-1
    An alternative implementation of Linux sound support
local/alsa-utils 1.1.4-1
    An alternative implementation of Linux sound support
local/archlinux-keyring 20170611-1
    Arch Linux PGP keyring
local/archlinux-menus 2.5-3
    Arch Linux specific XDG-compliant menu
local/autofs 5.1.2-1
    A kernel-based automounter for Linux.
local/avahi 0.6.32-3
    Service Discovery for Linux using mDNS/DNS-SD -- compatible with Bonjour
local/bumblebee 3.2.1-15
    NVIDIA Optimus support for Linux through VirtualGL
local/cgroup_event_listener 4.12-1 (linux-tools)
    Simple listener of cgroup events
local/cpupower 4.12-1 (linux-tools)
    Linux kernel tool to examine and tune power saving related features of your
    processor
local/filesystem 2017.03-2 (base)
    Base Arch Linux files
local/hdparm 9.52-1
    A shell utility for manipulating Linux IDE drive/driver parameters
local/i2c-tools 3.1.1-2
    Heterogeneous set of I2C tools for Linux that used to be part of lm-sensors
local/i7z 0.27.2.git1-4
    A better i7 (and now i3, i5) reporting tool for Linux
local/iptables 1.6.1-1
    Linux kernel packet control tool
local/kdenlive 17.04.3-1 (kde-applications kdemultimedia)
    A non-linear video editor for Linux using the MLT video framework
local/keyutils 1.5.10-1
    Linux Key Management Utilities
local/kmod 24-1
    Linux kernel module management tools and library
local/ladspa 1.13-6
    Linux Audio Developer's Simple Plugin API (LADSPA)
local/lib32-alsa-lib 1.1.4.1-1
    An alternative implementation of Linux sound support (32 bit)
local/lib32-keyutils 1.5.10-1
    Linux Key Management Utilities (32-bit)
local/lib32-libva 1.8.2-1
    Video Acceleration (VA) API for Linux (32-bit)
local/lib32-libxshmfence 1.2-1
    a library that exposes a event API on top of Linux futexes (32-bit)
local/lib32-util-linux 2.29.2-1
    Miscellaneous system utilities for Linux (32-bit)
local/libaio 0.3.110-1
    The Linux-native asynchronous I/O facility (aio) library
local/libiec61883 1.2.0-4
    A higher level API for streaming DV, MPEG-2 and audio over Linux IEEE 1394
local/libimobiledevice 1.2.0-6
    Library that talks the protocols to support iPhone and iPod Touch devices on
    Linux
local/libraw1394 2.1.2-1
    Provides an API to the Linux IEEE1394 (FireWire) driver
local/libtraceevent 4.12-1 (linux-tools)
    Linux kernel trace event library
local/libutil-linux 2.29.2-2
    util-linux runtime libraries
local/libva 1.8.2-1
    Video Acceleration (VA) API for Linux
local/libxshmfence 1.2-1
    a library that exposes a event API on top of Linux futexes
local/linux 4.11.9-1 (base)
    The Linux kernel and modules
local/linux-api-headers 4.10.1-1
    Kernel headers sanitized for use in userspace
local/linux-ck-skylake 4.11.11-1 (ck-skylake)
    The Linux-ck-skylake kernel and modules with the ck2 patchset featuring MuQSS
    CPU scheduler v0.156 Sixth Gen Intel Core i3/i5/i7 optimized.
local/linux-ck-skylake-headers 4.11.11-1 (ck-skylake)
    Header files and scripts for building modules for Linux-ck-skylake kernel Sixth
    Gen Intel Core i3/i5/i7 optimized.
local/linux-firmware 20170422.ade8332-1
    Firmware files for Linux
local/linux-headers 4.11.9-1
    Header files and scripts for building modules for Linux kernel
local/linux-lts 4.9.37-1
    The Linux-lts kernel and modules
local/linux-lts-headers 4.9.37-1
    Header files and scripts for building modules for Linux-lts kernel
local/lirc 1:0.9.4.d-1
    Linux Infrared Remote Control utilities
local/man-pages 4.11-1 (base)
    Linux man pages
local/mdadm 4.0-1 (base)
    A tool for managing/monitoring Linux md device arrays, also known as Software
    RAID
local/mplayer 37916-2
    Media player for Linux
local/net-tools 1.60.20160710git-1
    Configuration tools for Linux networking
local/nvidia-dkms 381.22-4
    NVIDIA driver sources for linux
local/pacman-mirrorlist 20170714-1
    Arch Linux mirror list for use by pacman
local/perf 4.12-1 (linux-tools)
    Linux kernel performance auditing tool
local/phoronix-test-suite 7.2.0-1
    The most comprehensive testing and benchmarking platform available for Linux
local/ratslap 0.2.0-1
    Community made Linux drivers for Logitech mice (Currently only G300s)
local/thermald 1.6-1
    The Linux Thermal Daemon program from 01.org
local/tmon 4.12-1 (linux-tools)
    Monitoring and Testing Tool for Linux kernel thermal subsystem
local/turbostat 4.12-1 (linux-tools)
    Report processor frequency and idle statistics
local/usbip 4.12-1 (linux-tools)
    An USB device sharing system over IP network
local/util-linux 2.29.2-2 (base base-devel)
    Miscellaneous system utilities for Linux
local/v4l-utils 1.12.5-1
    Userspace tools and conversion library for Video 4 Linux
local/x86_energy_perf_policy 4.12-1 (linux-tools)
    Read or write MSR_IA32_ENERGY_PERF_BIAS

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 94
Model name:            Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz
Stepping:              3
CPU MHz:               1052.435
CPU max MHz:           3200.0000
CPU min MHz:           800.0000
BogoMIPS:              4608.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              6144K
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

The system is in my sig.
I took a picture of a common error message. Sorry for the terrible quality.

Microcode is installed. AFAIK I did not forget to mount anything as it's done automatically, or maybe I did not understand the question.

Last edited by OdinEidolon (2017-07-18 13:37:48)

R00KIE · 2017-07-18 13:16:17

Where did you get that 4.11.11-1-ck-skylake kernel?

OdinEidolon · 2017-07-18 13:18:09

R00KIE wrote:

Where did you get that 4.11.11-1-ck-skylake kernel?

Linux-ck. But as I said, this happens even with the vanilla and the LTS kernel.

seth · 2017-07-18 13:35:06

The uploaded image seems empty?

OdinEidolon · 2017-07-18 13:38:01

seth wrote:

The uploaded image seems empty?

Thanks, fixed now.

seth · 2017-07-18 13:45:23

That's more a problem with the init process, ie. likely systemd?

pacman -Qi systemd libsystemd
pacman -Qkk systemd libsystemd

OdinEidolon · 2017-07-18 13:47:51

seth wrote:

That's more a problem with the init process, ie. likely systemd?
pacman -Qi systemd libsystemd
pacman -Qkk systemd libsystemd

It might be, however why would it be solved the way I solved it? The first time I had this problem it reported a problem with libdl, but I was not able to reproduce it and get a screenshot.

EDIT: I stress that I can reproduce the problem by re-installing glibc. Then I have to start from the liveUSB and copy the mentioned libraries.

pacman -Qi systemd libsystemd
Name            : systemd
Version         : 233.75-3
Description     : system and service manager
Architecture    : x86_64
URL             : https://www.github.com/systemd/systemd
Licenses        : GPL2  LGPL2.1
Groups          : None
Provides        : nss-myhostname  systemd-tools=233.75  udev=233.75
Depends On      : acl  bash  cryptsetup  dbus  iptables  kbd  kmod  hwids  libcap
                  libgcrypt  libsystemd  libidn  lz4  pam  libelf  libseccomp
                  util-linux  xz
Optional Deps   : libmicrohttpd: remote journald capabilities
                  quota-tools: kernel-level quota management
                  systemd-sysvcompat: symlink package to provide sysvinit binaries
                  [installed]
                  polkit: allow administration as unprivileged user [installed]
Required By     : accountsservice  bluez-utils  chromium  colord  dhcpcd
                  lib32-systemd  libinput  libmbim  libpulse  libwacom
                  media-player-info  mkinitcpio  modemmanager  netctl  pcmciautils
                  polkit  prelink-systemd  profile-sync-daemon  rtkit  subversion
                  systemd-sysvcompat  udisks  upower
Optional For    : None
Conflicts With  : nss-myhostname  systemd-tools  udev
Replaces        : nss-myhostname  systemd-tools  udev
Installed Size  : 17.51 MiB
Packager        : Christian Hesse <arch@eworm.de>
Build Date      : Thu 06 Jul 2017 15:27:15 CEST
Install Date    : Sat 08 Jul 2017 14:36:17 CEST
Install Reason  : Installed as a dependency for another package
Install Script  : Yes
Validated By    : Signature

Name            : libsystemd
Version         : 233.75-3
Description     : systemd client libraries
Architecture    : x86_64
URL             : https://www.github.com/systemd/systemd
Licenses        : GPL2
Groups          : None
Provides        : libsystemd.so=0-64  libudev.so=1-64
Depends On      : glibc  libcap  libgcrypt  lz4  xz
Optional Deps   : None
Required By     : dbus  device-mapper  dhcpcd  libatasmart  libgudev  libudev0-shim
                  libusb  lvm2  mariadb  procps-ng  python-systemd  rpcbind
                  systemd  usbip  util-linux  xorg-server-xvfb
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 1514.00 KiB
Packager        : Christian Hesse <arch@eworm.de>
Build Date      : Thu 06 Jul 2017 15:27:27 CEST
Install Date    : Sat 08 Jul 2017 14:36:12 CEST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

pacman -Qkk systemd libsystemd
backup file: systemd: /etc/systemd/journald.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/journald.conf (Size mismatch)
backup file: systemd: /etc/systemd/logind.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/logind.conf (Size mismatch)
backup file: systemd: /etc/systemd/system.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/system.conf (Size mismatch)
warning: systemd: /usr/share/polkit-1/rules.d/systemd-networkd.rules (Permission denied)
systemd: 1402 total files, 1 altered file
warning: libsystemd: /usr/lib/libsystemd.so.0.18.0 (Size mismatch)
warning: libsystemd: /usr/lib/libudev.so.1.6.6 (Size mismatch)
libsystemd: 12 total files, 2 altered files

Last edited by OdinEidolon (2017-07-18 13:49:04)

seth · 2017-07-18 13:55:56

COUGH

warning: libsystemd: /usr/lib/libsystemd.so.0.18.0 (Size mismatch)
warning: libsystemd: /usr/lib/libudev.so.1.6.6 (Size mismatch)

Try init=/bin/bash

OdinEidolon · 2017-07-18 13:59:11

seth wrote:

COUGH

warning: libsystemd: /usr/lib/libsystemd.so.0.18.0 (Size mismatch)
warning: libsystemd: /usr/lib/libudev.so.1.6.6 (Size mismatch)

Try init=/bin/bash

Sorry, what does that mean exactly and why should it be related to glibc? Pardon my ignorance!

EDIT: (I'll try later)

Last edited by OdinEidolon (2017-07-18 13:59:28)

seth · 2017-07-18 14:03:13

It's a kernel parameter, see https://wiki.archlinux.org/index.php/Kernel_parameters
It'll turn bash into your init process, effectively gettign you some sort of busybox.

The suspicion is that the newer glibc is binary incompatible with whatever you did to your libsystemd, effectively breaking the init process (w/o being the actual culprit, but only the trigger)

OdinEidolon · 2017-07-18 14:04:58

seth wrote:

It's a kernel parameter, see https://wiki.archlinux.org/index.php/Kernel_parameters
It'll turn bash into your init process, effectively gettign you some sort of busybox.
The suspicion is that the newer glibc is binary incompatible with whatever you did to your libsystemd, effectively breaking the init process (w/o being the actual culprit, but only the trigger)

Yeah, my question was related to the first part, not to the init= part. If this is true how I'd fix it? I guess reinstalling the systemd packages would be enough?

EDIT: doing a

sudo pacman -Qkk | grep -i warning

I get TONS of those warnings...

Last edited by OdinEidolon (2017-07-18 14:07:07)

seth · 2017-07-18 14:07:12

Yup. But ensure the resulting files are clean, thus first try to figure what happened wrong.
Any idea why there's a size mismatch in those libraries? Are they maybe symlinks?

Edit: Warnings are not critical per se, eg. it's "normal" to have altered config files etc. Sometimes a wrong timestamp because you touched the file or whatever.
But in this case you've a "bad" file for a core library. That should raise a red flag.

Last edited by seth (2017-07-18 14:09:00)

OdinEidolon · 2017-07-18 14:08:52

seth wrote:

Yup. But ensure the resulting files are clean, thus first try to figure what happened wrong.
Any idea why there's a size mismatch in those libraries? Are they maybe symlinks?

I have this warning for a TON of packages, it seems. I use prelink, could it be a possible culprit, linking libraries?

Trilby · 2017-07-18 14:10:36

(EDIT: whoa, I'm slow. This was a reply to post #13)

The pacman output means just what it says: there is a mismatch in the size of those library files. It's not clear how that would happen (my guess might be some odd sym linking or renaming of lib files) but regardless of the cause, those library files are not actually what they are supposed to be. How would it be related to glibc ... that's also not clear, but if some libs were symlinked/renamed then you could have systemd library files that are actually linked against the older glibc and cannot run with the updated version.

Regardless, the problem is clearly that those are the wrong systemd library files, so init=/bin/bash will take systemd out of the equation and should give you a working shell.

seth · 2017-07-18 14:11:30

Did you see the big fat red WARNING on top of the prelink wiki page?
https://wiki.archlinux.org/index.php/Prelink

OdinEidolon · 2017-07-18 14:18:09

seth wrote:

Did you see the big fat red WARNING on top of the prelink wiki page?
https://wiki.archlinux.org/index.php/Prelink

Maybe! Who remembers what happened when I installed the system... I had used prelink for several years previously, it did seem to speedup my system a little bit (placebo?).
I'll remove prelink and see what's up.

OdinEidolon · 2017-07-18 14:28:39

I removed prelinking, rebooted, checked -Qkk (now I have only a few tens of warnings instead of several hundreds), reinstalled glibc, rebooted, and it seems to work. I'll report back if any problem arises but for now I'll mark this as solved.

Thanks for the help everybody, especially seth. Sick troubleshooting skills there (also from Trilby!).

Last edited by OdinEidolon (2017-07-18 14:32:55)

R00KIE · 2017-07-18 15:13:27

Just to be on the safe side I would reinstall the packages that own those libraries that pacman still warns about, otherwise you may have other weird hard to debug problems later on.

OdinEidolon · 2017-07-18 15:34:11

R00KIE wrote:

Just to be on the safe side I would reinstall the packages that own those libraries that pacman still warns about, otherwise you may have other weird hard to debug problems later on.

That's good advice.

eschwartz · 2017-07-19 00:27:35

https://www.reddit.com/r/archlinux/comm … ks_system/

lol, reddit strikes again.

The good news is that bugwrangler #2 (who hasn't had nearly as much time to become bitter and jaded like #1) merely mistook an AUR package modifying large numbers of repo-installed files for a partial update, but was still correct that it was a user support question suitable for the forums rather than a bug in glibc.

Arch Linux

#1 2017-07-18 12:06:39

[SOLVED] glibc update bricks system

#2 2017-07-18 12:23:24

Re: [SOLVED] glibc update bricks system

#3 2017-07-18 12:26:11

Re: [SOLVED] glibc update bricks system

#4 2017-07-18 12:50:25

Re: [SOLVED] glibc update bricks system

#5 2017-07-18 13:04:40

Re: [SOLVED] glibc update bricks system

#6 2017-07-18 13:16:17

Re: [SOLVED] glibc update bricks system

#7 2017-07-18 13:18:09

Re: [SOLVED] glibc update bricks system

#8 2017-07-18 13:35:06

Re: [SOLVED] glibc update bricks system

#9 2017-07-18 13:38:01

Re: [SOLVED] glibc update bricks system

#10 2017-07-18 13:45:23

Re: [SOLVED] glibc update bricks system

#11 2017-07-18 13:47:51

Re: [SOLVED] glibc update bricks system

#12 2017-07-18 13:55:56

Re: [SOLVED] glibc update bricks system

#13 2017-07-18 13:59:11

Re: [SOLVED] glibc update bricks system

#14 2017-07-18 14:03:13

Re: [SOLVED] glibc update bricks system

#15 2017-07-18 14:04:58

Re: [SOLVED] glibc update bricks system

#16 2017-07-18 14:07:12

Re: [SOLVED] glibc update bricks system

#17 2017-07-18 14:08:52

Re: [SOLVED] glibc update bricks system

#18 2017-07-18 14:10:36

Re: [SOLVED] glibc update bricks system

#19 2017-07-18 14:11:30

Re: [SOLVED] glibc update bricks system

#20 2017-07-18 14:18:09

Re: [SOLVED] glibc update bricks system

#21 2017-07-18 14:28:39

Re: [SOLVED] glibc update bricks system

#22 2017-07-18 15:13:27

Re: [SOLVED] glibc update bricks system

#23 2017-07-18 15:34:11

Re: [SOLVED] glibc update bricks system

#24 2017-07-19 00:27:35

Re: [SOLVED] glibc update bricks system

Board footer