Kernel panic: system won't boot

loh · 2018-06-27 12:05:00

Hello there,

I am unable to boot my laptop following a pacman update yesterday. After the syslinux splash I immediately get a kernel panic. I can't post the full trace (although I could attach a screen shot) but it starts with a segfault and ends with reference to a potential page fault.

Since I am unable to reboot into a root shell I've been using a usb arch iso to mount the root and boot partitions and then examine the system using arch-chroot. Having looked at the pacman log, there was no update to the kernel, or any kernel modules, prior to the issue occurring. When I first mounted /boot the partition was empty except for a lost+found directory. After further attempts it is now populated with the linux and linux-fallback images, but no syslinux directory (although I still see my regular syslinux splash and menu options on boot).

So far I've tried the following, all with no success:

- Rebuild the ramdisk using mkinitcpio:
The "autodetect" module fails to detect the root partition, but it continues and seems to finish without error.

- Downgrade the kernel using the pacman cache:
This reports a segfault in some /tmp/alpm_????/.INSTALL script, then seemingly continues as normal (I can't find the file afterwards to see why or where the segfault is occuring). The same thing happens if I update the kernel again, or delete from the cache and re-download the latest kernel version.

- Run fschk on all partitions:
No problems.

- Run a MemTest overnight using the live usb:
No errors.

- Tried reinstalling syslinux:
Since the mounted boot partition didn't have a syslinux directory, attempting to use the syslinux-install_update script failed. To get anywhere I needed to reinstall syslinux using pacman (creating the syslinux directory myself wasn't good enough), then run the script again. The script now runs, but errors with yet another segfault, stating the it "Could not find filesystem on / (root) or /boot." Despite the segfault, there is now a syslinux.cfg file in the boot partition (this is the only thing in the syslinux directory). Bizarrely, although this is different to my old config file (since I use a splash, and have added a MemTest entry), when I exit the chroot, unmount and reboot, I still see the original menu screen, including the splash image (which is located inside of the boot partition!). I've not yet tried a full manual re-install of syslinux.

Does anyone have any suggestion as to what might be going wrong? I'm happy to post more information as needed. I'm wondering if my SSD is on the blink (this laptop is pretty old) and something has messed up in the boot partition. The system is fully backed up so it's not much hassle to reinstall if necessary.

For reference: the laptop is a ThinkPad x301, i.e. fairly standard (but old) hardware. I've been running the same Arch install on this machine (just updating) for over eight years with no issues.

Cheers.

Last edited by loh (2018-06-27 13:11:17)

seth · 2018-06-27 13:37:12

Yeah, knowing which module segfaults would probably be helpful… https://bbs.archlinux.org/viewtopic.php?id=57855
The typical issue is that you forgot to mount the /boot partition when updating (thus mkinitcpio operates on a dir in the root partition and the kernel booted from the /boot partition differs from the installed one)

loh · 2018-06-27 14:19:20

I tried rebooting a few more times and on occasion it produces a shorter trace for the Kernel panic that doesn't disappear off the top of the screen. Here's me copying the full trace by hand:

starting version 238
/dev/sda3: recovering journal
/dev/sda3: clean, 534998/1966080 files, 6340684/7864320 blocks
[    3.118505] systemd[1]: segfault at 8 ip 00007f08ca7a13bf sb 00007fffd7574000 error 4 in ld-2.27.so[7f08ca791000+2500]
[    3.118536] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    3.118536]
[    3.118600] CPU: 1 PID: 1 Comm: systemd Not tainted 4.17.2-1-ARCH #1
[    3.118646] Hardware name: LENOVO 2777CTO/2777CTO, BIOS 6EET47WW (3.07 ) 10/13/2009
[    3.118690] Call Trace:
[    3.118719]  dump_stack+0x5c/0x80
[    3.118747]  panic+0xe7/0x247
[    3.118773]  do_exit.cold.4+0x1e/0xd3
[    3.118802]  do_group_exit+0x33/0xa0
[    3.118831]  get_signal+0x24d/0x5d0
[    3.118859]  ? page_fault+0x8/0x30
[    3.118888]  do_signal+0x36/0x610
[    3.118914]  ? force_sig_info_fault+0x9b/0xf0
[    3.118948]  ? __bad_area_nosemaphore.cold.9+0x5f/0x66
[    3.118985]  ? page_fault+0x8/0x30
[    3.119013]  exit_to_usermode_loop+0x85/0xa0
[    3.119045]  prepare_exit_to_usermode+0x64/0x90
[    3.119077]  retint_user+0x8/0x8
[    3.119104] RIP: 0033:0x7f08ca7a13bf
[    3.119130] RSP: 002b:00007fffd7574000 EFLAGS: 0001020206
[    3.119166] RAX: 0000000000000000 RBX: 000055fdd8221e58 RCX: 0000000000000000
[    3.119211] RDX: 0000000000000001 RSI: 00007f08ca959d20 RDI: 000055fdd821b1e5
[    3.119255] RBP: 00007f08ca959890 R08: 0000000000000000 R09: 0000000000000005
[    3.119299] R10: fffffffffffff245 R11: 0000000000000000 R12: 00007fffd7574fbb
[    3.119344] R13: 0000000003a775e9 R14: 00007f08ca9b7100 R15: 0000000000000011
[    3.119425] Kernel Offset: 0x8000000 from 0xfffffffff81000000 (relocation range: 0xfffffff800000-0xfffffffbfffffff)
[    3.119501] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x000000b
[    3.119501]  ]---

It refers to ld-2.27.so, so perhaps glibc is corrupted (although it wasn't in the list of pacman installed packages prior to the problem and was last updated back in early May). I'll try to reinstall that and see what happens.

The typical issue is that you forgot to mount the /boot partition when updating (thus mkinitcpio operates on a dir in the root partition and the kernel booted from the /boot partition differs from the installed one)

The boot partition is definitely mounted.

Last edited by loh (2018-06-27 14:25:13)

loh · 2018-06-27 14:32:56

Still no joy having re-installed glibc (after deleting the cached version).

When using arch-chroot I get the following during some of the post-install systemd hooks:

Skipped: Current root is not booted.

Last edited by loh (2018-06-27 14:36:34)

seth · 2018-06-27 15:11:58

Fyi, you can of course post screenshots if you *really* had to hand-copy the text (there're also screenreaders for this)

Was systemd among the updated packages?

loh · 2018-06-27 15:29:55

Fyi, you can of course post screenshots if you *really* had to hand-copy the text (there're also screenreaders for this)

Thanks, I'll remember this for next time. Thank goodness I can touch type :-)

Was systemd among the updated packages?

No. The list of packages that were updated prior to the Kernel panic are:

dwarffortress
ethtool
expac
ilmbase
gimp
graphicsmagick
libmagick
imagemagick
jupyter-widgetsnbextension
lic32-attr
lib32-acl
lib32-mesa
libmagick6
nm-connection-editor
network-manager-applet
offlineimap
php
php-apache
python-ipywidgets
sysfsutils
qt5-webengine
source-highlight
unrar
wine
youtube-dl

The last time the kernel was updated was in the upgrade previous to this, after which I could boot successfully.

seth · 2018-06-27 16:00:56

The only package in that list with a remote cance to break the system is sysfsutils…

loh · 2018-06-27 16:26:23

Agreed. Sadly downgrading / reinstalling sysfsutils doesn't fix the problem.

loh · 2018-06-27 16:51:46

On other boots I sometimes see a slightly different Kernel panic trace. This one starts with:

WARNING: CPU: 1 PID: 1 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40

seth · 2018-06-27 17:02:14

The sysfsutils impact would be indirect - did you mkinitcpio after the downgrade?

loh · 2018-06-27 17:28:17

Yes, I rebuilt the initramfs after the downgrade. When running mkinitcpio in the arch-chroot I always see this when the autodetect module is loaded:

ERROR: failed to detect root filesystem

One thing I realised is that I was suppressing some output from the early boot stage. The kernel panic occurs just after

Running cleanup hook [udev]

loqs · 2018-06-27 17:53:54

Can you chroot in from boot media and post the output of `pacman -Qkk` see if anything unexpected has become corrupted.

loh · 2018-07-02 07:47:45

There was nothing untoward in the output. I ended up reinstalling since it was quicker than trying to figure out the problem and everything has been working fine since. This was my first unrecoverable Arch upgrade issue in close to 10 years, so I can't complain too much :-)

Arch Linux

#1 2018-06-27 12:05:00

Kernel panic: system won't boot

#2 2018-06-27 13:37:12

Re: Kernel panic: system won't boot

#3 2018-06-27 14:19:20

Re: Kernel panic: system won't boot

#4 2018-06-27 14:32:56

Re: Kernel panic: system won't boot

#5 2018-06-27 15:11:58

Re: Kernel panic: system won't boot

#6 2018-06-27 15:29:55

Re: Kernel panic: system won't boot

#7 2018-06-27 16:00:56

Re: Kernel panic: system won't boot

#8 2018-06-27 16:26:23

Re: Kernel panic: system won't boot

#9 2018-06-27 16:51:46

Re: Kernel panic: system won't boot

#10 2018-06-27 17:02:14

Re: Kernel panic: system won't boot

#11 2018-06-27 17:28:17

Re: Kernel panic: system won't boot

#12 2018-06-27 17:53:54

Re: Kernel panic: system won't boot

#13 2018-07-02 07:47:45

Re: Kernel panic: system won't boot

Board footer