[solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

fieito · 2024-01-20 22:56:22

I've a Lenovo Thinkpad T470 and had been running the linux-zen kernel 6.7.zen3-1 since last Monday with no issues.

I updated the system on Thursday, which brought in the version 37.2-1 of mkinitcpio and since then I can't boot any 6.7 kernel. I first experienced this with linux-zen but the same happens with the regular linux kernel. LTS is the only one that currently boots and I can't find out why.

So far, what I've been able to establish is that whenever I try to boot a 6.7 kernel the boot process stalls on the message:

Starting Authorization Manager...

After that, the lighs on the mute buttons for speakers and microphone keep on and the CapsLock LED keeps blinking. After reading this forum it seems this is a kernel panic, however I don't know why and haven't been able to find out.

This is what I've tried:

Re-running mkinitcpio to generate a new UKI
Adding the KVM hook to mkinitcpio.conf
Adding i915.i915_enable_rc6=0 to the kernel command line. This was after finding someone that had had a similar error a few years ago. No luck
Adding systemd.journald.forward_to_console=1 console=tty1 to the kernel command line. According to the wiki this should show the reasons for panic on the early boot. No extra info.

I'm not sure where to look anymore. Has anyone had a similar issue? Would anyone be able to point me to where I can try and find out what's happening?

Last edited by fieito (2024-01-23 00:07:12)

fieito · 2024-01-20 23:11:12

As an update, I've tried downgrading to the previous 2 versions of mkinitcpio (37.1 and 36.1) with no improvements.

I also forgot to mention that I had also tried downgrading to the previous version of the linux-firmware package, thinking that the issue might be there. No luck either.

Head_on_a_Stick · 2024-01-20 23:20:32

Sounds like https://bbs.archlinux.org/viewtopic.php?id=291893 & https://bbs.archlinux.org/viewtopic.php?id=291943

Can you try a kernel bisect? Forum member loqs linked to some resources that might be useful in the second thread.

V1del · 2024-01-20 23:21:44

If the capslock leds are blinking then you have a kernel panic and an actual bug in the kernel

After producing the problem and rebooting into the LTS kernel, what do you get from

sudo journalctl -b-1

https://wiki.archlinux.org/title/List_o … n_services

fieito · 2024-01-20 23:35:31

Hi thanks for the replies.

Will get onto responding to your questions ASAP. In the meantime I tried something else to make sure the issue was in the kernel and I've been able to boot OK by downgrading to the previous version of the linux-zen kernel (6.6.10.zen1-1) which seems to indicate to me that this is a kernel issue...

I'll try and reproduce the issue again with the linux kernel and post back the journalctl output as V1del mentioned.

fieito · 2024-01-21 00:00:24

V1del wrote:

If the capslock leds are blinking then you have a kernel panic and an actual bug in the kernel
After producing the problem and rebooting into the LTS kernel, what do you get from
sudo journalctl -b-1
https://wiki.archlinux.org/title/List_o … n_services

Here's what that shows: https://bin.bloat.cat/?93fe555436646397 … H8BDtZ1RTw

I think there's stuff missing from there as it doesn't include the "Starting Authorization Manager" which is the line where it all freezes (except for the caps lock blinking) and I don't know why.

seth · 2024-01-21 10:09:06

I think there's stuff missing from there as it doesn't include the "Starting Authorization Manager"

'cause you're rebooting w/ the power button
=> https://wiki.archlinux.org/title/Kdump

You can also try and see whether https://wiki.archlinux.org/title/Genera … l_messages gets you some interesting messages, but you'll have to link photos of the monitor w/ that (please don't embed them)

Jan 20 23:45:20 archlinux kernel: Command line: root=UUID=XXX rw add_efi_memmap rd.luks.name=AAA=corvocrs rd.luks.options=AAA=tpm2-device=auto rd.luks.options=discard 
    lsm=landlock,lockdown,yama,integrity,apparmor,bpf audit=1 audit_backlog_limit=8192 module.sig_enforce=1 splash vt.global_cursor_default=0 
    intel_iommu=on fbcon=nodefer slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none page_alloc.shuffle=1
    randomize_kstack_offset=on extra_latent_entropy debugfs=off oops=panic lockdown=confidentiality random.trust_cpu=off
    random.trust_bootloader=off efi=disable_early_pci_dma iommu.passthrough=0 iommu.strict=1 zswap.enabled=0 systemd.journald.forward_to_console=1 console=tty1

And I'd blindly try to simplify on this, there're quite some lockdown related errors/warnings, iommu settings are always suspicious - basically cut out all the hardening attempts and see whether it boots. If so, add them one-by-one or bisecting unitl you've found the culprit.

fieito · 2024-01-23 00:05:40

seth wrote:

And I'd blindly try to simplify on this, there're quite some lockdown related errors/warnings, iommu settings are always suspicious - basically cut out all the hardening attempts and see whether it boots. If so, add them one-by-one or bisecting unitl you've found the culprit.

I thought I'd try this first because, as soon as I read it, I thought: "I should have done that first!". I guess my theory was: this set of arguments worked until now, so it should work.

I have just gone through about 15 reboots by reducing my command line arguments to a minimum & then adding one by one (2 and 3 at once close to the end, tbh) and found out that if I included:

extra_latent_entropy debugfs=off

Then it wouldn't start. I'm still not sure where the issue was exactly. This time with the debug text & systemd forwarding everything it complained about 3 things: tlp, NetworkManager & Authorization Manager. After a little while, it just stayed on "A start job is running for Authorization Manager" and it kept repeating.

See picture of the screen with the issue caused by those two arguments here.

Hopefully this is at least helpful for folks having issues like this.

Thanks everyone for your help, it's much appreciated!!

Last edited by fieito (2024-01-23 00:08:59)

seth · 2024-01-23 09:34:03

You could try "debugfs=no-mount"
extra_latent_entropy will certainly slow down the boot but I cannot come up with an explanation why it'd break - maybe add "transparent_hugepage=never" (there're related issues w/ 6.7 and THP has always been a source of trouble)

fieito · 2024-01-23 19:01:11

seth wrote:

You could try "debugfs=no-mount"
extra_latent_entropy will certainly slow down the boot but I cannot come up with an explanation why it'd break - maybe add "transparent_hugepage=never" (there're related issues w/ 6.7 and THP has always been a source of trouble)

Adding these, even with THP, makes it stall again as it did before...

seth · 2024-01-23 22:13:54

Jan 20 23:45:20 archlinux kernel: pinctrl core: failed to create debugfs directory
Jan 20 23:45:20 archlinux kernel: Could not create 'lock_event_counts' debugfs entries
Jan 20 23:45:20 archlinux kernel: zsmalloc: debugfs not available, stat dir not created
Jan 20 23:45:20 archlinux kernel: cec: Failed to create debugfs cec dir
Jan 20 23:45:27 corvo systemd[1]: Kernel Debug File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/debug).

Arch Linux

#1 2024-01-20 22:56:22

[solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#2 2024-01-20 23:11:12

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#3 2024-01-20 23:20:32

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#4 2024-01-20 23:21:44

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#5 2024-01-20 23:35:31

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#6 2024-01-21 00:00:24

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#7 2024-01-21 10:09:06

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#8 2024-01-23 00:05:40

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#9 2024-01-23 09:34:03

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#10 2024-01-23 19:01:11

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

#11 2024-01-23 22:13:54

Re: [solved] Kernel 6.7.0 doesn't boot after upgrading mkinitcpio

Board footer