[Solved] Glibc update causes kernel panic due to "no working init.. "

Lasse100 · 2017-09-11 18:20:21

Hello,

After updating yesterday, my laptop gave a kernel panic at startup because no working init could be found:

"Kernel panic: not syncing: no working init found"

After doing a binary search on the updated packages, this seems to be caused by the update of glibc from version 2.25-7 to 2.26-3.
This means that it is either caused by the update from 2.25 to 2.26 or by this patch:
https://git.archlinux.org/svntogit/pack … f5324bf080

Any help would be appreciated.

Last edited by Lasse100 (2017-09-14 08:44:50)

loqs · 2017-09-11 22:43:45

Welcome to the arch linux forums Lasse100.
glibc 2.26-3 is in core so if it caused all systems to fail to boot would expect many more reports on the issue.
The system now boots with all packages updated apart from glibc which is version 2.25-7? Is the system i686 or x86_64?
The panic is when the system loads the initrd?

mcmillan · 2017-09-12 00:04:45

After doing a binary search on the updated packages, this seems to be caused by the update of glibc from version 2.25-7 to 2.26-3.

As loqs says, the new glibc seems to be working fine on my system (and presumably most people's since this wasn't an issue I've noticed anybody else posting).

Can you post the results from your search that suggested this was the issue? It might point to something else that wasn't properly updated. Posting your pacman log also would let us know what was updated yesterday. My initial thought is either you are running partial updates or have some package from the AUR or other unofficial source that's causing the problem

Lasse100 · 2017-09-12 07:30:08

I find it strange as well. I always perform full updates. After the problem occurred on Sunday, I used the arch archive to revert back to the archive from 2017-09-08, which worked.
Then I started selectively upgrading and downgrading packages, of which the results did not make any sense at first. I first assumed that the obvious culprits should either be mkinitcpio, linux or linux-api-headers. But these three packages could be upgraded on their own without problems. Then I started running "mkinitcpio -p linux" after every upgrade or downgrade, which revealed that glibc was the problem.

I now have a working system with all packages fully updated, except for the following three packages:

binutils-2.29.0-1  glibc-2.26-3  lib32-glibc-2.26-2

Note that binutils and lib32-glibc depend on the new version of glibc and therefore cannot be updated.
Currently these packages are at these versions:

binutils 2.28.0-4 glibc 2.25-7 lib32-glibc 2.25-7

It seems that mkinitcpio uses the standard c library to compile something, which no longer works with the new version of glibc.

Some answers to your questions:
- The system is x86_64. Note that my computer is a rather old HP elitebook 8540w from 2010.
- The panic occurs immediately after the initial image is loaded, so that is presumably indeed when the system loads initrd.
- My pacman log is rather messy after my binary search experiments. The log from my update on Sunday can be found here:
https://pastebin.com/83Q3Bz87
The full pacman log can be found here (note that I sometimes use yaourt as a frontend, so you may be seeing some artifacts from that).
https://paste.ee/p/H3JCq
- I never run partial updates (except for the binary search I did yesterday)
- I do not run any custom kernels or anything like that. I do have some AUR packages, but these are all user-level packages that - in my mind - could never interfere with the kernel starting process:

acroread 9.5.5-7
btrbk 0.25.1-1
keepass-plugin-rpc 1:1.7.2-1
mathematica 11.1.1-1
mcrl2 201409.1-3
package-query 1.9-2
pamac-aur 5.1.1-3
prover9 2009.11A-2
spotify 1.0.59.395-1
swftools 0.9.2-5
yaourt 1.9-1
yices-bin 2.5.2-1

Does anyone know what the correct path for the init= option should be? Then I can try setting this path manually. I already tried /usr/bin/systemd/systemd and /usr/bin/bash, but both do not work.

loqs · 2017-09-12 11:41:14

init should be a file called init in the root of the initrd. You can extract the contents of the initrd with something like:

$ bsdcpio -iF /boot/initramfs-linux.img

Lone_Wolf · 2017-09-12 12:20:20

I have multiple init-systems installed, and use

init=/usr/lib/systemd/systemd

for the configurations where systemd needs to be PID 1

Lasse100 · 2017-09-12 13:35:59

I have done some further investigation, and the problem appears exactly when upgrading from 2.26-1 to 2.26-2. This corresponds to the following commit:
https://git.archlinux.org/svntogit/pack … 9de4d0ec65

As can be seen, the only changes made are the removal of the build parameter "--enable-kernel=2.6.32" and a move of upstream commit from
https://sourceware.org/git/?p=glibc.git … 39d3e18369
to
https://sourceware.org/git/?p=glibc.git … e7c0e2a999

At this point, I do not have any ideas for further investigations. Does anyone have a suggestion?

Lone_Wolf · 2017-09-12 14:01:47

What bootloader are you using ?

Lasse100 · 2017-09-12 15:01:12

I'm using Grub.

loqs · 2017-09-12 20:00:37

Would suggest adding the systemd hook plus other relevant changes to mkinitcpio.conf or removing it if you already use it.
This should switch the init being provided from mkinitcpio-busybox to systemd or vice-versa. If it still fails then both are affected and you may need to bisect glibc to find the cause.

digifuzzy · 2017-09-12 20:57:37

I think there may be some truth to Lasse100's observations.
I had valgrind installed which bumped the glibc version to latest (2.26-3).

After upgrading the kernel, I went to reboot and got stopped right at boot with the same "-2" error (init system not found)
I tried the suggestions in...
[SOLVED]Kernel panic init error
[Solved] Kernel panics with "No init found" after upgrade
...without success. I kept getting the same error message.

Only by downgrading glibc to 2.25-7 could I get the system boot restored.
I had to use live cd to:
- downgrade
- re-install filesystem grub mkinitcpio linux
- mkinitcpio -p linux
- grub-mkconfig -o /boot/grub/grub.cfg

After reboot, all was restored.

loqs · 2017-09-12 21:09:50

@digifuzzy if your system uses an Intel CPU are microcode updates enabled?
Unless you are performing partial updates your system should have updated to glibc 2.26-3 irrespective of valgrind 3.13.0-3 having a version dependency on glibc 2.26.
Unfortunately it is still not clear if the bug is in glibc 2.26-3 or it is triggering a latent issue in another package such as mkinitcpio-busybox.
You could file a bug report against glibc. Bisecting the issue first would help determine the cause.

digifuzzy · 2017-09-12 21:15:13

loqs wrote:

@digifuzzy if your system uses an Intel CPU are microcode updates enabled?
...
You could file a bug report against glibc. Bisecting the issue first would help determine the cause.

I'm using an AMD bulldozer cpu. I'll check.

loqs · 2017-09-12 22:25:52

On AMD systems the microcode update is still handled by the kernel itself assuming the linux-firmware package is installed.
So at least for digifuzzy that seems not to be the cause. Would suggest reporting it as a bug against glibc see if Bartłomiej Piotrowski has more insight into the issue.

digifuzzy · 2017-09-12 22:27:41

loqs wrote:

On AMD systems the microcode update is still handled by the kernel itself assuming the linux-firmware package is installed.

# pacman -Qs linux-firmware
local/linux-firmware 20170622.7d2c913-1

Lasse100 · 2017-09-13 13:40:40

I have been trying to do a bisect on the glibc source, but I do not really have the necessary skills for this.
I also opened the following ticket:
https://bugs.archlinux.org/task/55620

Lasse100 · 2017-09-14 08:42:24

This problem now seems to be solved by glibc 2.26-4. Thanks everyone for your suggestions.

Last edited by Lasse100 (2017-09-14 08:43:54)

Arch Linux

#1 2017-09-11 18:20:21

[Solved] Glibc update causes kernel panic due to "no working init.. "

#2 2017-09-11 22:43:45

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#3 2017-09-12 00:04:45

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#4 2017-09-12 07:30:08

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#5 2017-09-12 11:41:14

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#6 2017-09-12 12:20:20

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#7 2017-09-12 13:35:59

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#8 2017-09-12 14:01:47

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#9 2017-09-12 15:01:12

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#10 2017-09-12 20:00:37

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#11 2017-09-12 20:57:37

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#12 2017-09-12 21:09:50

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#13 2017-09-12 21:15:13

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#14 2017-09-12 22:25:52

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#15 2017-09-12 22:27:41

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#16 2017-09-13 13:40:40

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

#17 2017-09-14 08:42:24

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

Board footer