[Solved] stack in boot process at loading initramfs

jancici · 2022-08-15 19:21:00

I dont know that happen. I did reboot laptop today and boot process stack. Laptop thinkpad T470 is totaly frozen, I can only hold power button to poeroff it.

I can only see these 3 lines:

Loading ../vmlinuz-linux... ok
Loading ../intel-ucode.img...ok
Loading ../initramfs-linux...ok

I have nvme disk and this is mounts:
/dev/nvme0n1p2 /
/dev/nvme0n1p1 /boot

ls -la /boot
EFI
intel-ucode.img
vmlinuz-linux

ls -la /boot/EFI
arch
initramfs-linux-fallback.img
initramfs-linux.img
intel-ucode.img
syslinux
vmlinuz-linux

As you can see I am using syslinux as bootmanager. I think it is working right because I can see blue menu and boot process start.
this is my config for default option, fallback option stack too.

LABEL arch
MENU LABEL Arch Linux
LINUX ../vmlinuz-linux
APPEND root=/dev/nvme0n1p2 rw
INITRD ../intel-ucode.img,../initramfs-linux.img

I did boot from usb where I have fresh archlinux iso. I did bring network up and mount disks and run arch-chroot /mnt where is mounted my ssd.
I did reinstall linux (currently it is 5.19.1-arch2-1) and linux-firmware and intel-ucode.
Then I did run mkinitcpio -P

I am using /etc/mkinitcpio.d/linux.preset to copy necessary files to /boot/EFI.

Please, I did run out of ideas what can be wrong and how to fix this.
thanks for help.

Last edited by jancici (2022-09-08 17:19:12)

twelveeighty · 2022-08-15 19:45:54

Have you tried using the UUID= syntax instead of device names? APPEND root=UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

jancici · 2022-08-15 20:04:33

I did try and did not help :-(

jancici · 2022-08-15 20:09:28

I did also try to add debug parameter to kernel like this

APPEND root=/dev/nvme0n1p2 rw debug ignore_loglevel

but there was nothing more on screen

seth · 2022-08-15 20:25:17

https://bbs.archlinux.org/viewtopic.php?id=278856 - also syslinux, lts test pending
https://bbs.archlinux.org/viewtopic.php?id=278897 is on LVM but a kernel downgrade seems to have resolved it

=> Test a kernel downgrade or the the LTS kernel

perseus · 2022-08-15 21:13:33

Many thanks Seth,

That is very helpful, if only from the point of view of my sanity.

I made the rescue USB and, before (in despair) downgrading to 5.18.16, I used it to edit my syslinux.cfg - deleting everything bar the minimum in each stanza in an attempt to get the installed 5.19.1 to boot. No dice - same boot loop as before. I found a few similar examples on other forums from years ago - but of course such things do happen for different reasons. Those same edited stanzas now boot the downgraded kernel without problems.

I have been thinking of installing Grub or some other loader on the laptop, just out of interest, but no time for that today.

LTS remains possible, but (although I am not nearly so committed as I used to be), I have been full-time on linux for work and play for > 20 years now, and have run every kernel since whatever minor of v.2 was then current - I hate to give up using the latest version.

seth · 2022-08-15 21:26:32

I hate to give up using the latest version.

LTS was just an alternative to the downgrade to test whether it's the same 5.19 regression.

Ftr: are you using any kind of encryption on the root partition?

perseus · 2022-08-15 21:43:42

Seth wrote:

Ftr: are you using any kind of encryption on the root partition?

No, not on either machine - nor ever have.

jancici · 2022-08-16 06:17:54

thanks, downgrade kernel to 5.18.16-arch1-1 did help

it is not clear to me, what is problem with 5.19.1? is it bug? should I change / update my system (config)?

seth · 2022-08-16 06:41:18

There're now three of you, where syslinux fails to boot the 5.19.1 kernel - I don't think that's a local config issue, nor a coincidence.
Historically, there's https://bugzilla.kernel.org/show_bug.cgi?id=95841 - it could be that syslinux fails to decompress the specific kernel build or it could be a systematic issue.
In the latter case, the problem won't likely solve itself, so we'd ideally get better debug info - https://wiki.archlinux.org/title/Syslin … loader_log

vitorqb · 2022-08-16 07:24:37

I can confirm the issue using GRUB as well. After upgrading the kernel, it fail to decrypt the encrypted disk during boot (some message saying it couldn't find the disk(. I had to downgrade it from a USB installation:

[2022-08-16T09:09:32+0200] [ALPM] downgraded linux (5.19.1.arch2-1 -> 5.18.16.arch1-1)

After downgrading it works fine.

perseus · 2022-08-16 08:14:02

vitorqb wrote:

I can confirm the issue using GRUB as well

Thank you. That is useful. I would probably have switched to GRUB on my laptop. Of course it might work for me, but messing with boot loaders and frozen machines when I don't have a deep understanding of the technicalities is unlikely to produce much useful feedback.

seth · 2022-08-16 09:57:20

Hold your horses, at least for perseus we've established that disc encryption isn't related to the situation (and jancici doesn't seem to use an ecrypted partition or LVM either)
Between perseus and jancici we've nvme and sata, so it's also not likely to be a device driver (failing to init the drive)

vitorqb wrote:

some message saying it couldn't find the disk

The specifics here would actually be really helpful.
Is it the root device and does your grub.cfg reference it by UUID or some device name?
Make sure there's no "quiet" parameter and in doubt just link a photo of the error
Does adding "rootwait" to the kernel parameters help? (In doubt this will just hang forever because the drive never shows up)

Maniaxx · 2022-08-16 10:59:18

Maybe also try:
'earlyprintk=efi' or 'earlyprintk=vga'.

jancici · 2022-08-16 19:44:20

yes, I dont use encrypted partition and using nvme.
I did also try use UUID but did not help.

here is my APPEND from syslinux

APPEND root=/dev/nvme0n1p2 rw debug ignore_loglevel earlyprintk=vga

and try to follow https://wiki.archlinux.org/title/Syslin … loader_log so I did remove UI and setup PROMPT to 1

but nothing new, kernel 5.19.1.arch2-1 stack and there is no any new message on screen, only those 3 lines
Loading ../vmlinuz-linux... ok
Loading ../intel-ucode.img...ok
Loading ../initramfs-linux...ok

I did also install lts kernel so I can easily start on next boot.
so if you have any tip I could try I can but my time is bit limit these days.

seth · 2022-08-16 20:09:11

I did also try use UUID but did not help.

That was actually for vitorqb as I'm not convinced that he happens to be the only user on grub that suffers from the same problem that pretty much everyone w/ syslinux is currently facing.

BwK · 2022-08-16 20:48:05

seth wrote:

There're now three of you, where syslinux fails to boot the 5.19.1 kernel - I don't think that's a local config issue, nor a coincidence.
Historically, there's https://bugzilla.kernel.org/show_bug.cgi?id=95841 - it could be that syslinux fails to decompress the specific kernel build or it could be a systematic issue.
In the latter case, the problem won't likely solve itself, so we'd ideally get better debug info - https://wiki.archlinux.org/title/Syslin … loader_log

I tried to modify my syslog.cfg to get better debug info. Unfortunately there was nothing more to see. Just

../../vmlinuz-linux... ok
../../initramfs-linux.img... ok

as before. Recompiling syslog as suggested in the linked wiki failed, even in a clean chroot.

seth · 2022-08-17 06:42:53

https://bbs.archlinux.org/viewtopic.php … 8#p2052198

vitorqb · 2022-08-17 16:49:45

seth wrote:

Is it the root device and does your grub.cfg reference it by UUID or some device name?

My expertise here is quite limited but hopefully those are the relevant lines:

insmod part_gpt
insmod cryptodisk
insmod luks
insmod gcry_rijndael
insmod gcry_rijndael
insmod gcry_sha512
insmod lvm
insmod ext2
cryptomount -u 90c946319f7a4b32bd1458db1fc46d7d
set root='lvmid/zWg8xa-XI3n-K7Om-DcPV-69gW-nf8u-EvgLBc/2xHt9r-CXMv-YUzf-2AQt-Dc23-jdPG-1PHgVW'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='lvmid/zWg8xa-XI3n-K7Om-DcPV-69gW-nf8u-EvgLBc/2xHt9r-CXMv-YUzf-2AQt-Dc23-jdPG-1PHgVW'  72a0e494-a403-4575-868b-0e83be34980c
else
  search --no-floppy --fs-uuid --set=root 72a0e494-a403-4575-868b-0e83be34980c
fi

seth wrote:

Make sure there's no "quiet" parameter and in doubt just link a photo of the error
Does adding "rootwait" to the kernel parameters help? (In doubt this will just hang forever because the drive never shows up)

I really appreciate your help here, and I'm very sorry to say that I won't be able to reproduce the error today. The issue is in my work laptop so I need to be extra careful and reserve extra time to recover from it later. I'll post more info here as soon as I find the time to reproduce it.

seth wrote:

That was actually for vitorqb as I'm not convinced that he happens to be the only user on grub that suffers from the same problem that pretty much everyone w/ syslinux is currently facing.

True, it's weird that I'm the only GRUB user having issues. Let's see if I can reproduce it later.

Last edited by vitorqb (2022-08-17 16:50:41)

jancici · 2022-08-19 14:50:22

I did update to 5.19.2.arch1-1 today
but problem remains

loqs · 2022-08-19 16:06:28

You could try the bisection kernels from the thread seth linked and see if all of those have the issue.
https://bbs.archlinux.org/viewtopic.php … 6#p2052346

Last edited by loqs (2022-08-19 16:13:46)

jbg · 2022-08-20 13:34:50

This issue is reproducible on our AWS instances at my workplace. We use a minimal image built with pacstrap that boots via UEFI with syslinux. 5.18.x boots fine, 5.19.x reboots immediately after loading the initramfs. I will try with `earlyprintk` as we have serial console logs from the instances.

vitorqb · 2022-08-20 15:29:53

I can no longer reproduce the issue I had on boot w/ GRUB with the latest version (5.19.2-arch1-1).

jbg · 2022-08-20 15:45:36

earlyprintk reveals this just before the reboot:

Physical KASLR disabled: no suitable memory region!
Invalid physical address chosen!

And, to be clear, this is with 5.19.2-arch1-1 and syslinux.

Last edited by jbg (2022-08-20 15:46:06)

loqs · 2022-08-20 17:45:51

@jbg out of interest do you know what CPU the instance is hosted on?

Arch Linux

#1 2022-08-15 19:21:00

[Solved] stack in boot process at loading initramfs

#2 2022-08-15 19:45:54

Re: [Solved] stack in boot process at loading initramfs

#3 2022-08-15 20:04:33

Re: [Solved] stack in boot process at loading initramfs

#4 2022-08-15 20:09:28

Re: [Solved] stack in boot process at loading initramfs

#5 2022-08-15 20:25:17

Re: [Solved] stack in boot process at loading initramfs

#6 2022-08-15 21:13:33

Re: [Solved] stack in boot process at loading initramfs

#7 2022-08-15 21:26:32

Re: [Solved] stack in boot process at loading initramfs

#8 2022-08-15 21:43:42

Re: [Solved] stack in boot process at loading initramfs

#9 2022-08-16 06:17:54

Re: [Solved] stack in boot process at loading initramfs

#10 2022-08-16 06:41:18

Re: [Solved] stack in boot process at loading initramfs

#11 2022-08-16 07:24:37

Re: [Solved] stack in boot process at loading initramfs

#12 2022-08-16 08:14:02

Re: [Solved] stack in boot process at loading initramfs

#13 2022-08-16 09:57:20

Re: [Solved] stack in boot process at loading initramfs

#14 2022-08-16 10:59:18

Re: [Solved] stack in boot process at loading initramfs

#15 2022-08-16 19:44:20

Re: [Solved] stack in boot process at loading initramfs

#16 2022-08-16 20:09:11

Re: [Solved] stack in boot process at loading initramfs

#17 2022-08-16 20:48:05

Re: [Solved] stack in boot process at loading initramfs

#18 2022-08-17 06:42:53

Re: [Solved] stack in boot process at loading initramfs

#19 2022-08-17 16:49:45

Re: [Solved] stack in boot process at loading initramfs

#20 2022-08-19 14:50:22

Re: [Solved] stack in boot process at loading initramfs

#21 2022-08-19 16:06:28

Re: [Solved] stack in boot process at loading initramfs

#22 2022-08-20 13:34:50

Re: [Solved] stack in boot process at loading initramfs

#23 2022-08-20 15:29:53

Re: [Solved] stack in boot process at loading initramfs

#24 2022-08-20 15:45:36

Re: [Solved] stack in boot process at loading initramfs

#25 2022-08-20 17:45:51

Re: [Solved] stack in boot process at loading initramfs

Board footer