Boot problems after kernel update

mabox · 2024-09-21 11:01:02

Hello everyone,
I have the problem since a few updates now that when I update these packages the boot hangs at “Welcome to Arch Linux”. The screen seems to be freeze then..... if I turn it off a few times I can be have luck and it works ......
I am therefore now always back to the last version of these packages where the boot worked perfectly.
These are the packages that I have not updated since then.

linux
linux-firmware
linux-headers

Can anyone explain this? I am still on kernel version 6.10.2 for this reason :-(
The newest kernel is 6.10.10

Greetings
mabox

cryptearth · 2024-09-21 11:13:34

please give a brief list of your hardware (board, cpu, ram, gpu, additional components)

mabox · 2024-09-21 11:21:40

Hi cryptearth,
Thank you for your message.
I have this notebook: Tuxedo Aura Gen2 with 16GB RAM and AMD Ryzen 7 5700U
All technical infos can you see here, please scroll to technical details.
Tuxedo Aura Gen2

mabox · 2024-09-22 06:50:46

I have narrowed it down further, if I install the "linux-firmware" everything still works but if I install "linux" and "linux-headers" then there are problems.

Last edited by mabox (2024-09-22 06:53:37)

cryptearth · 2024-09-22 07:12:42

Hm, this machine looks kinda nice - purpose build for linux, nice connections - although just a 1MP cam for an office laptop sounds really bad, my very first digital cam had 1,3MP and anyone remembers Sony Cybershot with 5MP?
Anyway: Have you tried to boot without "quite" option to get more log output? You might add "debug" for even more log spam.
As you mention headers: Have you installed any self-built drivers or dkms? Could be these are not correctly updating along. Also: Can you boot the fallback option?

mabox · 2024-09-22 07:33:16

I think the notebook is great because it's specially made for Linux :-) I don't need a camera so it's ok.
I use systemd-boot loader and I can't find the “quit” setting anywhere. I think I'm already booting without this option.
How can I boot the fallback option? Unfortunately I don't know.

I am not consciously doing anything with self-built drivers and dkms. There are one or two packages from the AUR that I install via paru. e.g. also drivers for the notebook. But these are all already installed.
However, I have now noticed that paru no longer works after the update.... some libalm error. I then recompiled paru and installed linux and linux-headers..... After a reboot it has now worked, twice already..... strange.
Here is the output of what happens when I install linux and linux-headers.

core/linux          6.10.2.arch1-1  6.10.10.arch1-1           0,04 MiB
core/linux-headers  6.10.2.arch1-1  6.10.10.arch1-1           0,02 MiB


Pre-transaction hooks are started
(1/2) Removing linux initcpios...
(2/2) Remove DKMS modules
==> dkms remove --no-depmod tuxedo-drivers/4.6.1 -k 6.10.2-arch1-1
==> depmod 6.10.2-arch1-1


(1/2) Updated linux                                                              [#####################################################] 100%
(2/2) Updated linux-headers                                                      [#####################################################] 100%
:: Post-transaction hooks are started …
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod tuxedo-drivers/4.6.1 -k 6.10.10-arch1-1
==> depmod 6.10.10-arch1-1
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img
==> Starting build: '6.10.10-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [microcode]
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
  -> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
  -> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.10.10-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [microcode]
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
==> WARNING: Possibly missing firmware for module: 'ast'
  -> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
  -> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful

But as mentioned, it suddenly works now, all available updates are installed, both via pacman and paru.

EDIT: Oh no, it doesn't work after all... now suddenly it no longer boots.... I had to revert to the old kernel. Strange that it two times works and then no longer
i would like to upload a picture of the boot process here but it doesn't work right?

Last edited by mabox (2024-09-22 07:50:03)

Lone_Wolf · 2024-09-22 12:13:59

==> dkms remove --no-depmod tuxedo-drivers/4.6.1 -k 6.10.2-arch1-1

==> dkms install --no-depmod tuxedo-drivers/4.6.1 -k 6.10.10-arch1-1

You do use dkms for the tuxedo drivers, but you're lagging 4 versions behind as latest is 4.7.0-2 .

Update tuxedo-drivers-dkms and enable notifications for it so you'll get an email when the package is updated.
(you need to be logged in to AUR to use notifications)

mabox · 2024-09-22 17:26:06

@Lone_Wolf
Thanks for the tip about the AUR packages, I'd better keep an eye on them. The Tuxedo package is now also up to date, but nothing has changed.
As soon as I install linux and linux-headers the boot gets stuck at the “Welcome to Arch Linux”. What else do you think could be the problem? With kernel 6.10.2 everything is ok.

Head_on_a_Stick · 2024-09-22 18:04:24

Does the LTS kernel work?

cryptearth · 2024-09-22 18:29:53

mabox wrote:

stuck at the “Welcome to Arch Linux”

I don't get any such message when booting my system - using grub and standard kernel with only the root= option
please post your boot config and fstab as well as output of mount command - to me this sounds like some sort of boot splash which hides some error message - hence I recommended to remove the "quite" option - and any other like "splash" or "plymouth" - and maybe add "debug"
also: where's your esp mounted? /boot? /boot/efi? /efi? if /boot: is it correctly mount so that the new kernel and initrd gets installed onto it?
best guess: when correctly booting with 6.10.2 /boot doesn't get mounted and when you update the new kernel and initrd gets placed in /boot on the root partition instead of the esp - and you have something like "quite" "splash" or similar which hides an important error message which hints to what's the actual issue here is

mabox · 2024-09-23 07:54:13

@Head_on_a_Stick
Yes lts kernel works but i think that was to be expected as it is older than the 6.10.2 which still works.

Explanation. I have a BTRFS filesystem with 3 snapshots Testing, Stable, Oldstable.... I always work in "Testing" and if something goes wrong I can simply boot into my stable snapshot in the boot menu and have a working system again. To test the update again, I then simply delete the “Testing” snapshot and create a new snapshot of “Stable” for “Testing”. I have been doing this for over 2 years now and there have never been any problems.
I think "quite" is not active for me because when it works I can see the boot process but unfortunately not after the kernel update.
With this system and this configuration I have been updating the system for over 2 years now without any problems. I can't imagine that suddenly something should be mounted differently..... I will update again and then try to check what exactly happens before I boot.

Here are my boot config from systemd-boot and my esp is mounted in /boot/efi/

[root@tux /boot]# cat loader/entries/arch.conf 
title archlinux testing
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=LABEL=ARCH rootflags=subvol=@snapshots/TESTING rw


[root@tux /boot]# cat loader/loader.conf 
#timeout 3
#console-mode keep
default arch.conf
timeout 4
editor 0

Here is my “mount” output when I boot from the working kernel.

[root@tux /boot]# mount | grep TEST
/dev/nvme0n1p2 on / type btrfs (rw,noatime,compress=lzo,ssd,discard=async,space_cache=v2,subvolid=361,subvol=/@snapshots/TESTING)

cryptearth · 2024-09-23 08:45:40

I don't have any experience with root on btrfs.
Anxway - when does this "Welcome to Arch" come up? Do you get ANYthing between selecting the testing environment and the point when it hangs?
Have you examined the actual state of the snapshot after the update - i.e. check if 6.10.10 is both installed in the system and the version booted? How does it look when you arch-chroot into it? Can you even successfully boot a current arch install with a more recent kernel?
I still struggle hardly with this "Welcome to Arch" - where does this cone from? when in the boot process does it come up? with this default boot line you should see the services and targets comming up like mounting root, starting network ... etc

mabox · 2024-09-23 15:00:43

Thank you for making the effort anyway.
I am booted in my testing snapshot with kernel 6.10.2. In it I update to kernel 6.10.10 and then perform a reboot.
I see my systemd-boot selection menu, when I then choose "Testing" it hangs after "Welcome to Arch ...".
When I select "Stable" or "Oldstable" then everyting is fine, I see also "Welcome to Arch.... " but after that the detailed boot process as you describe, mounting root, starting network ...etc. Only "Testing" with kernel 6.10.10 stops.
So in principle I get past my boot loader configuration or my boot loader entries until “Welcome to Arch ...”.
I will do the update again later and check then before the reboot the kernel files in /boot..... Maybe I found something...

Head_on_a_Stick · 2024-09-23 18:20:33

Use the `file` command to compare the version of vmlinuz-linux on the EFI system partition with /usr/lib/modules/, the two should match.

mabox · 2024-09-24 15:09:25

It looks like I've solved it.

First I checked the two files with `file` like "Head_on_a_Stick" wrote but they were the same.

By the way the directory shows like this:

[root@tux ~]# ll /usr/lib/modules/
insgesamt 0
drwxr-xr-x 1 root root    184 24. Sep 16:47 ./
drwxr-xr-x 1 root root 149304 24. Sep 16:11 ../
drwxr-xr-x 1 root root     14 25. Okt 2022  6.0.1-arch2-1/
drwxr-xr-x 1 root root     14 29. Okt 2022  6.0.2-arch1-1/
drwxr-xr-x 1 root root     14  1. Nov 2022  6.0.5-arch1-1/
drwxr-xr-x 1 root root     14  5. Nov 2022  6.0.6-arch1-1/
drwxr-xr-x 1 root root     14 11. Nov 2022  6.0.7-arch1-1/
drwxr-xr-x 1 root root    524 24. Sep 16:47 6.10.10-arch1-1/
drwxr-xr-x 1 root root    500 23. Sep 07:42 6.6.52-1-lts/

I notice that 6.10.2 is missing and 6.10.10 is now in its place. Why is there still 6.0.1* there? Could I just delete it?

Ok, so I started a recherche of systemd-boot to see how I could get more information when booting. I search for any debug option.
But then I found in point 3.2.1 the update command.
systemd-boot

I just run `bootclt update` and this file was updated then:

[root@tux ~]# ll /boot/EFI/systemd/
-rwxr-xr-x 1 root root 105472 10. Sep 17:06 systemd-bootx64.efi*

The file in `/boot/EFI/BOOT/BOOTX64.EFI` was "skipped" because the command means that this file is already the newest version.

So after this “update” I update linux and linux-firmware again and I could already boot 3x with success.....
I have never updated the bootloader before but it seems like I need to do it more often or automate it.

I hope that was the solution. I will now boot a few more times to test. Thank you all who have given me tips.

EDIT: That is so strange, after I was able to boot successfully 3 times, it is not working again, so that was not the solution. I despair.

EDIT: After 5 more attempts, it has now booted again.

Last edited by mabox (2024-09-24 16:02:37)

V1del · 2024-09-24 16:26:59

This randomness might point to using non deterministic naming in your bootloader and/or fstab. Can you post your bootloader config/kernel cmdline and fstab?

mabox · 2024-09-24 19:12:15

@V1del
Thank you very much for this hint.
I actually had an error in the /etc/fstab. I have no idea how this could happen.
Ok but as summarize, my main problem was solved with bootclt update as descripe above.
This last problem now was due to an error in my fstab. Thank you all!

cryptearth · 2024-09-24 19:27:56

mabox wrote:

Ok but as summarize, my main problem was solved with bootclt update as descripe above.
This last problem now was due to an error in my fstab. Thank you all!

hence I requested you to post it

Arch Linux

#1 2024-09-21 11:01:02

Boot problems after kernel update

#2 2024-09-21 11:13:34

Re: Boot problems after kernel update

#3 2024-09-21 11:21:40

Re: Boot problems after kernel update

#4 2024-09-22 06:50:46

Re: Boot problems after kernel update

#5 2024-09-22 07:12:42

Re: Boot problems after kernel update

#6 2024-09-22 07:33:16

Re: Boot problems after kernel update

#7 2024-09-22 12:13:59

Re: Boot problems after kernel update

#8 2024-09-22 17:26:06

Re: Boot problems after kernel update

#9 2024-09-22 18:04:24

Re: Boot problems after kernel update

#10 2024-09-22 18:29:53

Re: Boot problems after kernel update

#11 2024-09-23 07:54:13

Re: Boot problems after kernel update

#12 2024-09-23 08:45:40

Re: Boot problems after kernel update

#13 2024-09-23 15:00:43

Re: Boot problems after kernel update

#14 2024-09-23 18:20:33

Re: Boot problems after kernel update

#15 2024-09-24 15:09:25

Re: Boot problems after kernel update

#16 2024-09-24 16:26:59

Re: Boot problems after kernel update

#17 2024-09-24 19:12:15

Re: Boot problems after kernel update

#18 2024-09-24 19:27:56

Re: Boot problems after kernel update

Board footer