Can not boot after update to linux 5.16

isaac_le · 2022-01-18 12:49:28

when I updated to 5.16 (today is 5.16.1) it got stuck at "Loading initial ramdisk" without any journalctl log

tail -n 500 /var/log/pacman.log

http://ix.io/3MDt (I boot 5.16 at 11:43):

journalctl --since 11:00

and then I switched from bootloader to systemd, the tty give me some message: http://ix.io/3MDq

I can boot Ok when rolling back to 5.15.13.

for the question from: https://bbs.archlinux.org/viewtopic.php … 9#p2016399

do you have earlier messages than that?

I turned on kernel debug:
http://ix.io/3MFf
http://ix.io/3MFg
http://ix.io/3MFh

sorry I don't know how to catch these log, after chroot, there are no /var/log/dmesg and nothing in journalctl about that boot

the LTS kernel still boots fine?

if you mean the 5.15.13, yes I can boot fine with it.

Head_on_a_Stick · 2022-01-18 15:16:27

isaac_le wrote:

I turned on kernel debug:
http://ix.io/3MFf
http://ix.io/3MFg
http://ix.io/3MFh

Those links are full of non-ASCII characters, for me at least.

But anyway do you have AMD graphics? See https://bbs.archlinux.org/viewtopic.php?id=273100

seth · 2022-01-18 15:40:28

They're images - the error is somewhere on the SATA.
Usually I'd say "cable", but that doesn't fit the kernel update condition.

if you mean the 5.15.13

No, the lts kernel: https://archlinux.org/packages/core/x86_64/linux-lts/
(as I assume "rolling back" means a complete downgrade and that doesn't tell us much about the culprit)

Try passing "libata.force=noncq" to the kernel (it gets disabled for "excessive errors" but maybe that's toolate™"
Also check the dmesg/journal w/ a "good" kernel to see whether the drive produces errors there as well (and they're just handled more gracefully)

isaac_le · 2022-01-19 10:04:48

Head_on_a_Stick wrote:

But anyway do you have AMD graphics?

No, i'm using intel nuc n3700

Try passing "libata.force=noncq" to the kernel

http://ix.io/3MKn
http://ix.io/3MKo
-----
for tls:

the lts kernel: https://archlinux.org/packages/core/x86_64/linux-lts/

`tail -n 500 /var/log/pacman.log`: http://ix.io/3MKq
Stuck with normal boot: http://ix.io/3MKp

Boot success full with kernel parameter `debug`:
- `journalctl -b` (3MB): http://sprunge.us/Jry1EV
- `dmesg`: http://ix.io/3MKr

seth · 2022-01-19 13:40:39

So the LTS blocks (at pretty much the same point as the main kernel) but boots w/ the debug parameter???
…
Does the main kernel (now) boot w/ "debug"?
Do they maybe both boot w/o?
Race condition? Loose connection? Temperature?

isaac_le · 2022-01-19 14:57:02

So the LTS blocks (at pretty much the same point as the main kernel) but boots w/ the debug parameter???

Yes, that's it.

Does the main kernel (now) boot w/ "debug"?
Do they maybe both boot w/o?

Sadly no.

I disassembled my PC and assembled them back but not help, really don't know what happened.
I think I will stick with to 5.15 for now
And may be the future kernel will be better with my hardware.

seth · 2022-01-19 15:02:22

If you replace "debug" w/ "boot_delay=100" (this delays every boot message by 100ms), do you still get the error?

isaac_le · 2022-01-19 15:43:04

boot_delay=100

TLS: it works. boot time is about 3 mins.
- `dmesg` http://sprunge.us/2zTBMM
- `journalctl -b` http://sprunge.us/IfKxk0

Main kernel: not working

Last edited by isaac_le (2022-01-19 15:43:59)

loqs · 2022-01-19 20:34:23

The links below are for a kernel built from the mid point of 5.15 to 5.16 is it good or bad?
https://drive.google.com/file/d/1l67bHh … sp=sharing linux-5.15.r7513.g2219b0ceefe8-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EHW_gc … sp=sharing linux-headers-5.15.r7513.g2219b0ceefe8-1-x86_64.pkg.tar.zst

seth · 2022-01-19 21:06:22

In the debug enabled boot, the µcode is patched right before the SSD is added, in the delayed boot right after.
Because of the proximity and the raciness and on a hunch: https://wiki.archlinux.org/title/Microc … ly_loading

isaac_le · 2022-01-20 05:20:58

loqs wrote:

The links below are for a kernel built from the mid point of 5.15 to 5.16 is it good or bad?
https://drive.google.com/file/d/1l67bHh … sp=sharing linux-5.15.r7513.g2219b0ceefe8-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EHW_gc … sp=sharing linux-headers-5.15.r7513.g2219b0ceefe8-1-x86_64.pkg.tar.zst

not working, stuck right after boot menu selection. the message same as: http://ix.io/3MFg

seth wrote:

In the debug enabled boot, the µcode is patched right before the SSD is added, in the delayed boot right after.
Because of the proximity and the raciness and on a hunch: https://wiki.archlinux.org/title/Microc … ly_loading

For TLS: I just notice that if I wait at the boot menu > 3s, it will boot successfully: w/ or w/o Microcode, w/o boot_dealy, w/o debug.

For 5.16 with Microcode: not working even waiting at the boot menu

loqs · 2022-01-20 05:47:26

Next bisection step
https://drive.google.com/file/d/1i9BVFk … sp=sharing linux-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Dfz0UP … sp=sharing linux-headers-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-20 07:11:57

loqs wrote:

Next bisection step
https://drive.google.com/file/d/1i9BVFk … sp=sharing linux-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Dfz0UP … sp=sharing linux-headers-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst

not working

# pacman -U linux-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst linux-headers-5.15.r4566.gfc02cb2b37fe-1-x86_64.pkg.tar.zst 
loading packages...
warning: downgrading package linux (5.15.r7513.g2219b0ceefe8-1 => 5.15.r4566.gfc02cb2b37fe-1)
warning: downgrading package linux-headers (5.16.1.arch1-1 => 5.15.r4566.gfc02cb2b37fe-1)
resolving dependencies...
looking for conflicting packages...

Packages (2) linux-5.15.r4566.gfc02cb2b37fe-1  linux-headers-5.15.r4566.gfc02cb2b37fe-1

Total Installed Size:  299.24 MiB
Net Upgrade Size:       -1.21 MiB

:: Proceed with installation? [Y/n] y
(2/2) checking keys in keyring                                    [####################################] 100%
(2/2) checking package integrity                                  [####################################] 100%
(2/2) loading package files                                       [####################################] 100%
(2/2) checking for file conflicts                                 [####################################] 100%
(2/2) checking available disk space                               [####################################] 100%
:: Running pre-transaction hooks...
(1/2) Removing linux initcpios...
(2/2) Remove DKMS modules
:: Processing package changes...
(1/2) downgrading linux                                           [####################################] 100%
(2/2) downgrading linux-headers                                   [####################################] 100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod r8168/8.049.02 -k 5.15.0-1-04566-gfc02cb2b37fe
==> depmod 5.15.0-1-04566-gfc02cb2b37fe
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.15.0-1-04566-gfc02cb2b37fe
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
Decompress:  1/23 files. Current: ...xhci-pci.ko.zst : 0 B...    ==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.15.0-1-04566-gfc02cb2b37fe
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
Decompress: 548/626 files. Current: ...nel/ext4.ko.zst : 384 KiB...    ==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

loqs · 2022-01-20 08:03:28

Bisecting: 2251 revisions left to test after this (roughly 11 steps)
https://drive.google.com/file/d/1YddxXS … sp=sharing linux-5.15rc7.r2443.gb7b98f868987-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1ZT-uBw … sp=sharing linux-headers-5.15rc7.r2443.gb7b98f868987-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-20 08:30:50

loqs wrote:

Bisecting: 2251 revisions left to test after this (roughly 11 steps)
https://drive.google.com/file/d/1YddxXS … sp=sharing linux-5.15rc7.r2443.gb7b98f868987-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1ZT-uBw … sp=sharing linux-headers-5.15rc7.r2443.gb7b98f868987-1-x86_64.pkg.tar.zst

it works!

loqs · 2022-01-20 16:26:01

git bisect good
Bisecting: 1127 revisions left to test after this (roughly 10 steps)
[6e5772c8d9cf0a77ba4d6fd34fd4126fb66c9983] Merge tag 'x86_cc_for_v5.16_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
https://drive.google.com/file/d/14mMxUg … sp=sharing linux-5.15.r1124.g6e5772c8d9cf-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K8YmhU … sp=sharing linux-headers-5.15.r1124.g6e5772c8d9cf-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-20 17:49:57

loqs wrote:

git bisect good
Bisecting: 1127 revisions left to test after this (roughly 10 steps)
[6e5772c8d9cf0a77ba4d6fd34fd4126fb66c9983] Merge tag 'x86_cc_for_v5.16_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
https://drive.google.com/file/d/14mMxUg … sp=sharing linux-5.15.r1124.g6e5772c8d9cf-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K8YmhU … sp=sharing linux-headers-5.15.r1124.g6e5772c8d9cf-1-x86_64.pkg.tar.zst

not working.

loqs · 2022-01-20 18:12:33

git bisect bad
Bisecting: 564 revisions left to test after this (roughly 9 steps)
[19901165d90fdca1e57c9baa0d5b4c63d15c476a] Merge tag 'for-5.16/inode-sync-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/1x9ZwO1 … sp=sharing linux-5.15.r559.g19901165d90f-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1s3DwAb … sp=sharing linux-headers-5.15.r559.g19901165d90f-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-21 03:10:24

loqs wrote:

git bisect bad
Bisecting: 564 revisions left to test after this (roughly 9 steps)
[19901165d90fdca1e57c9baa0d5b4c63d15c476a] Merge tag 'for-5.16/inode-sync-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/1x9ZwO1 … sp=sharing linux-5.15.r559.g19901165d90f-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1s3DwAb … sp=sharing linux-headers-5.15.r559.g19901165d90f-1-x86_64.pkg.tar.zst

not working.

loqs · 2022-01-21 15:39:49

git bisect bad
Bisecting: 281 revisions left to test after this (roughly 8 steps)
[33c8846c814c1c27c6e33af005042d15061f948b] Merge tag 'for-5.16/block-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/14vtP9V … sp=sharing linux-5.15.r277.g33c8846c814c-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1aNrDEm … sp=sharing linux-headers-5.15.r277.g33c8846c814c-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-21 16:54:35

loqs wrote:

git bisect bad
Bisecting: 281 revisions left to test after this (roughly 8 steps)
[33c8846c814c1c27c6e33af005042d15061f948b] Merge tag 'for-5.16/block-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/14vtP9V … sp=sharing linux-5.15.r277.g33c8846c814c-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1aNrDEm … sp=sharing linux-headers-5.15.r277.g33c8846c814c-1-x86_64.pkg.tar.zst

it works.

loqs · 2022-01-21 17:13:33

git bisect good
Bisecting: 161 revisions left to test after this (roughly 7 steps)
[643a7234e0960cf63f1a51a15cfc969fafcbabad] Merge tag 'for-5.16/drivers-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/19dLdiS … sp=sharing linux-5.15.r397.g643a7234e096-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Zsvn8c … sp=sharing linux-headers-5.15.r397.g643a7234e096-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-22 05:14:18

loqs wrote:

git bisect good
Bisecting: 161 revisions left to test after this (roughly 7 steps)
[643a7234e0960cf63f1a51a15cfc969fafcbabad] Merge tag 'for-5.16/drivers-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/19dLdiS … sp=sharing linux-5.15.r397.g643a7234e096-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Zsvn8c … sp=sharing linux-headers-5.15.r397.g643a7234e096-1-x86_64.pkg.tar.zst

It works.

loqs · 2022-01-22 05:32:28

git bisect good
Bisecting: 80 revisions left to test after this (roughly 6 steps)
[88459b50b42a4bd58e528006663afabd0b8652f2] io_uring: simplify io_file_supports_nowait()
https://drive.google.com/file/d/18pt3mF … sp=sharing linux-5.15rc6.r189.g88459b50b42a-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/13gc8-Q … sp=sharing linux-headers-5.15rc6.r189.g88459b50b42a-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-22 05:55:29

loqs wrote:

git bisect good
Bisecting: 80 revisions left to test after this (roughly 6 steps)
[88459b50b42a4bd58e528006663afabd0b8652f2] io_uring: simplify io_file_supports_nowait()
https://drive.google.com/file/d/18pt3mF … sp=sharing linux-5.15rc6.r189.g88459b50b42a-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/13gc8-Q … sp=sharing linux-headers-5.15rc6.r189.g88459b50b42a-1-x86_64.pkg.tar.zst

it works.

Arch Linux

#1 2022-01-18 12:49:28

Can not boot after update to linux 5.16

#2 2022-01-18 15:16:27

Re: Can not boot after update to linux 5.16

#3 2022-01-18 15:40:28

Re: Can not boot after update to linux 5.16

#4 2022-01-19 10:04:48

Re: Can not boot after update to linux 5.16

#5 2022-01-19 13:40:39

Re: Can not boot after update to linux 5.16

#6 2022-01-19 14:57:02

Re: Can not boot after update to linux 5.16

#7 2022-01-19 15:02:22

Re: Can not boot after update to linux 5.16

#8 2022-01-19 15:43:04

Re: Can not boot after update to linux 5.16

#9 2022-01-19 20:34:23

Re: Can not boot after update to linux 5.16

#10 2022-01-19 21:06:22

Re: Can not boot after update to linux 5.16

#11 2022-01-20 05:20:58

Re: Can not boot after update to linux 5.16

#12 2022-01-20 05:47:26

Re: Can not boot after update to linux 5.16

#13 2022-01-20 07:11:57

Re: Can not boot after update to linux 5.16

#14 2022-01-20 08:03:28

Re: Can not boot after update to linux 5.16

#15 2022-01-20 08:30:50

Re: Can not boot after update to linux 5.16

#16 2022-01-20 16:26:01

Re: Can not boot after update to linux 5.16

#17 2022-01-20 17:49:57

Re: Can not boot after update to linux 5.16

#18 2022-01-20 18:12:33

Re: Can not boot after update to linux 5.16

#19 2022-01-21 03:10:24

Re: Can not boot after update to linux 5.16

#20 2022-01-21 15:39:49

Re: Can not boot after update to linux 5.16

#21 2022-01-21 16:54:35

Re: Can not boot after update to linux 5.16

#22 2022-01-21 17:13:33

Re: Can not boot after update to linux 5.16

#23 2022-01-22 05:14:18

Re: Can not boot after update to linux 5.16

#24 2022-01-22 05:32:28

Re: Can not boot after update to linux 5.16

#25 2022-01-22 05:55:29

Re: Can not boot after update to linux 5.16

Board footer