You are not logged in.

#1 2017-09-11 18:20:21

Lasse100
Member
Registered: 2017-09-11
Posts: 6

[Solved] Glibc update causes kernel panic due to "no working init.. "

Hello,

After updating yesterday, my laptop gave a kernel panic at startup because no working init could be found:

"Kernel panic: not syncing: no working init found"

After doing a binary search on the updated packages, this seems to be caused by the update of glibc from version 2.25-7 to 2.26-3.
This means that it is either caused by the update from 2.25 to 2.26 or by this patch:
https://git.archlinux.org/svntogit/pack … f5324bf080

Any help would be appreciated.

Last edited by Lasse100 (2017-09-14 08:44:50)

Offline

#2 2017-09-11 22:43:45

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

Welcome to the arch linux forums Lasse100.
glibc 2.26-3 is in core so if it caused all systems to fail to boot would expect many more reports on the issue.
The system now boots with all packages updated apart from glibc which is version 2.25-7?  Is the system i686 or x86_64?
The panic is when the system loads the initrd?

Online

#3 2017-09-12 00:04:45

mcmillan
Member
Registered: 2006-04-06
Posts: 737

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

After doing a binary search on the updated packages, this seems to be caused by the update of glibc from version 2.25-7 to 2.26-3.

As loqs says, the new glibc seems to be working fine on my system (and presumably most people's since this wasn't an issue I've noticed anybody else posting).

Can you post the results from your search that suggested this was the issue? It might point to something else that wasn't properly updated. Posting your pacman log also would let us know what was updated yesterday. My initial thought is either you are running partial updates or have some package from the AUR or other unofficial source that's causing the problem

Offline

#4 2017-09-12 07:30:08

Lasse100
Member
Registered: 2017-09-11
Posts: 6

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I find it strange as well. I always perform full updates. After the problem occurred on Sunday, I used the arch archive to revert back to the archive from 2017-09-08, which worked.
Then I started selectively upgrading and downgrading packages, of which the results did not make any sense at first. I first assumed that the obvious culprits should either be mkinitcpio, linux or linux-api-headers. But these three packages could be upgraded on their own without problems. Then I started running "mkinitcpio -p linux" after every upgrade or downgrade, which revealed that glibc was the problem.

I now have a working system with all packages fully updated, except for the following three packages:

binutils-2.29.0-1  glibc-2.26-3  lib32-glibc-2.26-2

Note that binutils and lib32-glibc depend on the new version of glibc and therefore cannot be updated.
Currently these packages are at these versions:

binutils 2.28.0-4 glibc 2.25-7 lib32-glibc 2.25-7

It seems that mkinitcpio uses the standard c library to compile something, which no longer works with the new version of glibc.

Some answers to your questions:
- The system is x86_64. Note that my computer is a rather old HP elitebook 8540w from 2010.
- The panic occurs immediately after the initial image is loaded, so that is presumably indeed when the system loads initrd.
- My pacman log is rather messy after my binary search experiments. The log from my update on Sunday can be found here:
  https://pastebin.com/83Q3Bz87
  The full pacman log can be found here (note that I sometimes use yaourt as a frontend, so you may be seeing some artifacts from that).
  https://paste.ee/p/H3JCq
- I never run partial updates (except for the binary search I did yesterday)
- I do not run any custom kernels or anything like that. I do have some AUR packages, but these are all user-level packages that - in my mind - could never interfere with the kernel starting process:

acroread 9.5.5-7
btrbk 0.25.1-1
keepass-plugin-rpc 1:1.7.2-1
mathematica 11.1.1-1
mcrl2 201409.1-3
package-query 1.9-2
pamac-aur 5.1.1-3
prover9 2009.11A-2
spotify 1.0.59.395-1
swftools 0.9.2-5
yaourt 1.9-1
yices-bin 2.5.2-1

Does anyone know what the correct path for the init= option should be? Then I can try setting this path manually. I already tried /usr/bin/systemd/systemd and /usr/bin/bash, but both do not work.

Offline

#5 2017-09-12 11:41:14

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

init should be a file called init in the root of the initrd.  You can extract the contents of the initrd with something like:

$ bsdcpio -iF /boot/initramfs-linux.img

Online

#6 2017-09-12 12:20:20

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I have multiple init-systems installed, and use

init=/usr/lib/systemd/systemd

for the configurations where systemd needs to be PID 1


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#7 2017-09-12 13:35:59

Lasse100
Member
Registered: 2017-09-11
Posts: 6

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I have done some further investigation, and the problem appears exactly when upgrading from 2.26-1 to 2.26-2. This corresponds to the following commit:
https://git.archlinux.org/svntogit/pack … 9de4d0ec65

As can be seen, the only changes made are the removal of the build parameter "--enable-kernel=2.6.32" and a move of upstream commit from
https://sourceware.org/git/?p=glibc.git … 39d3e18369
to
https://sourceware.org/git/?p=glibc.git … e7c0e2a999

At this point, I do not have any ideas for further investigations. Does anyone have a suggestion?

Offline

#8 2017-09-12 14:01:47

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

What bootloader are you using ?


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#9 2017-09-12 15:01:12

Lasse100
Member
Registered: 2017-09-11
Posts: 6

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I'm using Grub.

Offline

#10 2017-09-12 20:00:37

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

Would suggest adding the systemd hook plus other relevant changes to mkinitcpio.conf or removing it if you already use it.
This should switch the init being provided from mkinitcpio-busybox to systemd or vice-versa.  If it still fails then both are affected and you may need to bisect glibc to find the cause.

Online

#11 2017-09-12 20:57:37

digifuzzy
Member
Registered: 2013-12-11
Posts: 12

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I think there may be some truth to Lasse100's observations.
I had valgrind installed which bumped the glibc version to latest (2.26-3).

After upgrading the kernel, I went to reboot and got stopped right at boot with the same "-2" error (init system not found)
I tried the suggestions in...
[SOLVED]Kernel panic init error
[Solved] Kernel panics with "No init found" after upgrade
...without success. I kept getting the same error message.

Only by downgrading glibc to 2.25-7 could I get the system boot restored.
I had to use live cd to:
- downgrade
- re-install filesystem grub mkinitcpio linux
- mkinitcpio -p linux
- grub-mkconfig -o /boot/grub/grub.cfg

After reboot, all was restored.

Offline

#12 2017-09-12 21:09:50

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

@digifuzzy if your system uses an Intel CPU are microcode updates enabled?
Unless you are performing partial updates your system should have updated to glibc 2.26-3 irrespective of valgrind 3.13.0-3 having a version dependency on glibc 2.26.
Unfortunately it is still not clear if the bug is in glibc 2.26-3 or it is triggering a latent issue in another package such as mkinitcpio-busybox.
You could file a bug report against glibc.  Bisecting the issue first would help determine the cause.

Online

#13 2017-09-12 21:15:13

digifuzzy
Member
Registered: 2013-12-11
Posts: 12

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

loqs wrote:

@digifuzzy if your system uses an Intel CPU are microcode updates enabled?
...
You could file a bug report against glibc.  Bisecting the issue first would help determine the cause.

I'm using an AMD bulldozer cpu. I'll check.

Offline

#14 2017-09-12 22:25:52

loqs
Member
Registered: 2014-03-06
Posts: 17,195

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

On AMD systems the microcode update is still handled by the kernel itself assuming the linux-firmware package is installed.
So at least for digifuzzy that seems not to be the cause.  Would suggest reporting it as a bug against glibc see if Bartłomiej Piotrowski has more insight into the issue.

Online

#15 2017-09-12 22:27:41

digifuzzy
Member
Registered: 2013-12-11
Posts: 12

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

loqs wrote:

On AMD systems the microcode update is still handled by the kernel itself assuming the linux-firmware package is installed.

# pacman -Qs linux-firmware
local/linux-firmware 20170622.7d2c913-1

Offline

#16 2017-09-13 13:40:40

Lasse100
Member
Registered: 2017-09-11
Posts: 6

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

I have been trying to do a bisect on the glibc source, but I do not really have the necessary skills for this.
I also opened the following ticket:
https://bugs.archlinux.org/task/55620

Offline

#17 2017-09-14 08:42:24

Lasse100
Member
Registered: 2017-09-11
Posts: 6

Re: [Solved] Glibc update causes kernel panic due to "no working init.. "

This problem now seems to be solved by glibc 2.26-4. Thanks everyone for your suggestions.

Last edited by Lasse100 (2017-09-14 08:43:54)

Offline

Board footer

Powered by FluxBB