Arch Linux

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

[SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Hi,

As usual when a new major kernel is about to be released, I give it a try when it's in the testing repos. It doesn't go well this time for 5.15 & 5.15.1, both stay stuck at the "Loading initial ramdisk ..." step, immediately before asking for my LUKS password. Facts:

- Machine is a usually unproblematic full-Intel Thinkpad T560, has been working for years on Arch. Until the 5.15 and 5.15.1 attempts, I was running 5.14.16 successfully. It is fully up-to date.

- No mkinitcpio error when installing the testing kernel. A few usual warnings for missing firmware, none that matters to me and would warrant grabbing them from AUR. See [mkinitcpio logs] at the end

- My `/etc/mkinitcpio.conf` has `HOOKS=(base udev autodetect keyboard modconf block encrypt filesystems fsck)` , and `MODULES=()` (both unchanged for years).

- I'm using GRUB, and LUKS encryption + ext4.

- Removing `quiet` (or switching to `noquiet`) from my Linux command doesn't provide any extra log

- Removing `/intel-ucode.img` from my initrd command doesn't change anything

- As a result to failing before asking for LUKS password, `journalctl -b` or `journalctl -b-1` has nothing to say about the failed boot.

Does this look like some misconfiguration of mine, or an actual new kernel bug (in which case I'll file a Flyspray bug)? I did my best to look for similar issues and provide all the typically requested debug info. Please ask if you need more, I'm available for more debug info.

Thanks for the halp.

==== [grub.cfg entry] ====

menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-3043e121-e400-484b-bd57-633e80577e43' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='hd0,msdos3'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 UUID_HERE
	else
	  search --no-floppy --fs-uuid --set=root UUID_HERE
	fi
	echo	'Loading Linux linux ...'
	linux	/vmlinuz-linux root=UUID=OTHER_UUID_HERE rw cryptdevice=/dev/sda4:cryptroot root=/dev/mapper/cryptroot loglevel=3 quiet cryptdevice=/dev/sda4:cryptroot root=/dev/mapper/cryptroot
	echo	'Loading initial ramdisk ...'
	initrd	/intel-ucode.img /initramfs-linux.img
}

==== [mkinitcpio logs] ====

:: Running post-transaction hooks...
(1/6) Arming ConditionNeedsUpdate...
(2/6) Updating module dependencies...
(3/6) Install DKMS modules
==> dkms install --no-depmod -m acpi_call -v 1.2.1 -k 5.15.1-arch1-1
==> depmod 5.15.1-arch1-1
(4/6) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.15.1-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.15.1-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

Last edited by ronjouch (2021-11-09 03:21:51)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,438

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

One thing that I see but that would normally lead to more visible errors is that you are using /dev/sda4 as the root device and /dev/sd names are not really stable and can change with just a single USB stick connected during boot: https://wiki.archlinux.org/title/Persis … ice_naming

Other than that hanging while waiting for an encryption relevant dialog might also be an issue of late crng maybe try the random.trust_cpu=on kernel parameter.

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

One thing that I see but that would normally lead to more visible errors is that you are using /dev/sda4 as the root device and /dev/sd names are not really stable and can change with just a single USB stick connected during boot: https://wiki.archlinux.org/title/Persis … ice_naming

Yeah, something I should improve, thanks for the reminder. This isn't the culprit, though, since other kernels (e.g. -lts)  generated from the same /etc/default/grub and grub-mkconfig boot fine.

Other than that hanging while waiting for an encryption relevant dialog might also be an issue of late crng maybe try the random.trust_cpu=on kernel parameter.

Just tested it, it didn't help, booting this kernel still hangs silently at "Loading initial ramdisk ..." with no extra error or output. Thanks for the suggestion.

One question/clarification, by the way: what does failure at this "Loading initial ramdisk ..." step mean? In particular, isn't it *sequential* to successfully "Loading Linux linux ..."? Thus, if step `initrd` is indeed sequential to step `linux` and `linux` seemed successful (which it is since I'm seeing followup step "Loading initial ramdisk ..."), why the suggestions -from you but other threads too- to adjust the *linux* command? Shouldn't we be working on instead adjusting the *initrd* command? What am I missing / misunderstanding?

Further suggestions and ideas to get more logs welcome.

Last edited by ronjouch (2021-11-07 20:44:10)

loqs

Member

Registered: 2014-03-06

Posts: 18,124

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Altering kernel behavior can effect any stage of the system booting / running.  Options passed from the bootloader to the kernel that are not directly used by the kernel are passed to the initrd.
Have you tried disabling KMS,  removing the quiet boot option and adding ignore_loglevel?

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Altering kernel behavior can effect any stage of the system booting / running.  Options passed from the bootloader to the kernel that are not directly used by the kernel are passed to the initrd.

Thanks; makes sense.

Have you tried disabling KMS

Just tried (Intel graphics, so as documented by your wiki article, I passed `nomodeset i915.modeset=0`), it didn't help.

removing the quiet boot option and adding ignore_loglevel?

Tried, I barely have more logs with them. With these options, I merely have one extra line (`Booting a command list`) when attempting to boot, bringing the total output to 3 lines:

  Booting a command list

Loading Linux linux ...
Loading initial ramdisk ...
_

This stays without moving for minutes (instead of showing the LUKS password prompt), and I end up rebooting with Ctrl+Alt+Del.

Last edited by ronjouch (2021-11-07 21:06:05)

loqs

Member

Registered: 2014-03-06

Posts: 18,124

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Try adding earlyprintk=efi,keep
If there is still no more output try iommu=soft acpi=off.

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Try adding earlyprintk=efi,keep

w00t, this enabled me to understand what's going on! Thaaaaaaaanks loqs, and thanks V1del !

- In older kernels, "Loading initial ramdisk ..." was followed by the LUKS prompt.

- In 5.15(.1), the LUKS prompt is absent, but it's actually visible in the `earlyprintk=efi,keep` logs, drowned among other logs and not visible unless I set the earlyprintk option! Then, if I type my LUKS password (to no apparent LUKS prompt! while the last line on screen is "Loading initial ramdisk ..."), the system boots! It feels like bad loglevel, or a race condition, or output not properly flushed to screen, or logs failing to be enabled before the LUKS prompt is written.

Marking as [SOLVED], and filed Flyspray bug https://bugs.archlinux.org/task/72658 .

Last edited by ronjouch (2021-11-07 22:06:46)

loqs

Member

Registered: 2014-03-06

Posts: 18,124

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Possibly the same framebuffer changed highlighted in https://bugs.archlinux.org/task/72645
Can you try building a kernel with those option changes reverted and see if that fixes the issue?

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Possibly the same framebuffer changed highlighted in https://bugs.archlinux.org/task/72645

Thx, I mentioned this possibility in Flyspray bug https://bugs.archlinux.org/task/72658 .

Can you try building a kernel with those option changes reverted and see if that fixes the issue?

Sure. Giving it a try and reporting back.

Last edited by ronjouch (2021-11-08 00:21:16)

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

Giving a custom kernel build a try and reporting back.

@loqs @V1del I have more news, that I detailed in https://bugs.archlinux.org/task/72658#comment203488 .

loqs

Member

Registered: 2014-03-06

Posts: 18,124

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

I reverted some more config changes

diff --git a/trunk/PKGBUILD b/trunk/PKGBUILD
index c53f6c5..e12c0d8 100644
--- a/trunk/PKGBUILD
+++ b/trunk/PKGBUILD
@@ -26,7 +26,7 @@ validpgpkeys=(
   'C7E7849466FE2358343588377258734B41C31549'  # David Runge <dvzrv@archlinux.org>
 )
 sha256sums=('SKIP'
-            '8a0bdb434ac26c16e3a96f127d68c5585f3c33ee9ed01e12a33e3a36889c9aeb')
+            'a4470c73cabe03a430a2841061ed51a9a414b71ea936a6cc7f2d5fc38a1237ac')
 
 export KBUILD_BUILD_HOST=archlinux
 export KBUILD_BUILD_USER=$pkgbase
diff --git a/trunk/config b/trunk/config
index 595a804..f3e2ec7 100644
--- a/trunk/config
+++ b/trunk/config
@@ -2258,12 +2258,13 @@ CONFIG_ISCSI_IBFT=m
 CONFIG_FW_CFG_SYSFS=m
 # CONFIG_FW_CFG_SYSFS_CMDLINE is not set
 CONFIG_SYSFB=y
-CONFIG_SYSFB_SIMPLEFB=y
+# CONFIG_SYSFB_SIMPLEFB is not set
 CONFIG_GOOGLE_FIRMWARE=y
 # CONFIG_GOOGLE_SMI is not set
 CONFIG_GOOGLE_COREBOOT_TABLE=m
 CONFIG_GOOGLE_MEMCONSOLE=m
 # CONFIG_GOOGLE_MEMCONSOLE_X86_LEGACY is not set
+CONFIG_GOOGLE_FRAMEBUFFER_COREBOOT=m
 CONFIG_GOOGLE_MEMCONSOLE_COREBOOT=m
 CONFIG_GOOGLE_VPD=m
 
@@ -4413,7 +4414,7 @@ CONFIG_I2C_MUX_MLXCPLD=m
 
 CONFIG_I2C_HELPER_AUTO=y
 CONFIG_I2C_SMBUS=m
-CONFIG_I2C_ALGOBIT=y
+CONFIG_I2C_ALGOBIT=m
 CONFIG_I2C_ALGOPCA=m
 
 #
@@ -5498,7 +5499,7 @@ CONFIG_IR_SERIAL_TRANSMITTER=y
 CONFIG_IR_SIR=m
 CONFIG_RC_XBOX_DVD=m
 CONFIG_IR_TOY=m
-CONFIG_CEC_CORE=y
+CONFIG_CEC_CORE=m
 CONFIG_CEC_NOTIFIER=y
 CONFIG_CEC_PIN=y
 CONFIG_MEDIA_CEC_RC=y
@@ -5749,7 +5750,8 @@ CONFIG_VIDEO_TW686X=m
 #
 CONFIG_VIDEO_IVTV=m
 CONFIG_VIDEO_IVTV_ALSA=m
-# CONFIG_VIDEO_FB_IVTV is not set
+CONFIG_VIDEO_FB_IVTV=m
+# CONFIG_VIDEO_FB_IVTV_FORCE_PAT is not set
 CONFIG_VIDEO_HEXIUM_GEMINI=m
 CONFIG_VIDEO_HEXIUM_ORION=m
 CONFIG_VIDEO_MXB=m
@@ -6226,7 +6228,7 @@ CONFIG_DVB_DUMMY_FE=m
 #
 # Graphics support
 #
-CONFIG_AGP=y
+CONFIG_AGP=m
 CONFIG_AGP_AMD64=m
 CONFIG_AGP_INTEL=m
 CONFIG_AGP_SIS=m
@@ -6235,13 +6237,12 @@ CONFIG_INTEL_GTT=m
 CONFIG_VGA_ARB=y
 CONFIG_VGA_ARB_MAX_GPUS=10
 CONFIG_VGA_SWITCHEROO=y
-CONFIG_DRM=y
+CONFIG_DRM=m
 CONFIG_DRM_MIPI_DBI=m
 CONFIG_DRM_MIPI_DSI=y
 CONFIG_DRM_DP_AUX_CHARDEV=y
-# CONFIG_DRM_DEBUG_MM is not set
 # CONFIG_DRM_DEBUG_SELFTEST is not set
-CONFIG_DRM_KMS_HELPER=y
+CONFIG_DRM_KMS_HELPER=m
 CONFIG_DRM_FBDEV_EMULATION=y
 CONFIG_DRM_FBDEV_OVERALLOC=100
 CONFIG_DRM_LOAD_EDID_FIRMWARE=y
@@ -6350,7 +6351,7 @@ CONFIG_DRM_ANALOGIX_DP=m
 CONFIG_DRM_BOCHS=m
 CONFIG_DRM_CIRRUS_QEMU=m
 CONFIG_DRM_GM12U320=m
-CONFIG_DRM_SIMPLEDRM=y
+CONFIG_DRM_SIMPLEDRM=m
 CONFIG_TINYDRM_HX8357D=m
 CONFIG_TINYDRM_ILI9225=m
 CONFIG_TINYDRM_ILI9341=m
@@ -6373,18 +6374,19 @@ CONFIG_DRM_PANEL_ORIENTATION_QUIRKS=y
 CONFIG_FB_CMDLINE=y
 CONFIG_FB_NOTIFY=y
 CONFIG_FB=y
-# CONFIG_FIRMWARE_EDID is not set
+CONFIG_FIRMWARE_EDID=y
+CONFIG_FB_BOOT_VESA_SUPPORT=y
 CONFIG_FB_CFB_FILLRECT=y
 CONFIG_FB_CFB_COPYAREA=y
 CONFIG_FB_CFB_IMAGEBLIT=y
-CONFIG_FB_SYS_FILLRECT=y
-CONFIG_FB_SYS_COPYAREA=y
-CONFIG_FB_SYS_IMAGEBLIT=y
+CONFIG_FB_SYS_FILLRECT=m
+CONFIG_FB_SYS_COPYAREA=m
+CONFIG_FB_SYS_IMAGEBLIT=m
 # CONFIG_FB_FOREIGN_ENDIAN is not set
-CONFIG_FB_SYS_FOPS=y
+CONFIG_FB_SYS_FOPS=m
 CONFIG_FB_DEFERRED_IO=y
-# CONFIG_FB_MODE_HELPERS is not set
-# CONFIG_FB_TILEBLITTING is not set
+CONFIG_FB_MODE_HELPERS=y
+CONFIG_FB_TILEBLITTING=y
 
 #
 # Frame buffer hardware drivers
@@ -6396,9 +6398,9 @@ CONFIG_FB_DEFERRED_IO=y
 # CONFIG_FB_ASILIANT is not set
 # CONFIG_FB_IMSTT is not set
 # CONFIG_FB_VGA16 is not set
-# CONFIG_FB_UVESA is not set
-# CONFIG_FB_VESA is not set
-# CONFIG_FB_EFI is not set
+CONFIG_FB_UVESA=m
+CONFIG_FB_VESA=y
+CONFIG_FB_EFI=y
 # CONFIG_FB_N411 is not set
 # CONFIG_FB_HGA is not set
 # CONFIG_FB_OPENCORES is not set
@@ -6429,10 +6431,11 @@ CONFIG_FB_DEFERRED_IO=y
 # CONFIG_FB_UDL is not set
 # CONFIG_FB_IBM_GXT4500 is not set
 # CONFIG_FB_VIRTUAL is not set
-# CONFIG_XEN_FBDEV_FRONTEND is not set
+CONFIG_XEN_FBDEV_FRONTEND=m
 # CONFIG_FB_METRONOME is not set
 # CONFIG_FB_MB862XX is not set
 # CONFIG_FB_HYPERV is not set
+CONFIG_FB_SIMPLE=m
 # CONFIG_FB_SSD1307 is not set
 # CONFIG_FB_SM712 is not set
 # end of Frame buffer Devices
@@ -6497,7 +6500,7 @@ CONFIG_DUMMY_CONSOLE_COLUMNS=80
 CONFIG_DUMMY_CONSOLE_ROWS=25
 CONFIG_FRAMEBUFFER_CONSOLE=y
 CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
-# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
+CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
 CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER=y
 # end of Console display driver support
 
@@ -8104,8 +8107,25 @@ CONFIG_DMABUF_HEAPS_CMA=y
 # end of DMABUF options
 
 CONFIG_DCA=m
-# CONFIG_AUXDISPLAY is not set
-# CONFIG_PANEL is not set
+CONFIG_AUXDISPLAY=y
+CONFIG_CHARLCD=m
+CONFIG_HD44780_COMMON=m
+CONFIG_HD44780=m
+CONFIG_KS0108=m
+CONFIG_KS0108_PORT=0x378
+CONFIG_KS0108_DELAY=2
+CONFIG_CFAG12864B=m
+CONFIG_CFAG12864B_RATE=20
+CONFIG_IMG_ASCII_LCD=m
+CONFIG_LCD2S=m
+CONFIG_PARPORT_PANEL=m
+CONFIG_PANEL_PARPORT=0
+CONFIG_PANEL_PROFILE=5
+# CONFIG_PANEL_CHANGE_MESSAGE is not set
+# CONFIG_CHARLCD_BL_OFF is not set
+# CONFIG_CHARLCD_BL_ON is not set
+CONFIG_CHARLCD_BL_FLASH=y
+CONFIG_PANEL=m
 CONFIG_UIO=m
 CONFIG_UIO_CIF=m
 CONFIG_UIO_PDRV_GENIRQ=m

Kernel built with the change above.  See if it helps
https://drive.google.com/file/d/1PuvBzd … sp=sharing linux-5.15.1.arch1-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K02MSo … sp=sharing linux-headers-5.15.1.arch1-1-x86_64.pkg.tar.zst

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

I reverted some more config changes [...] Kernel built with the change above.  See if it helps
https://drive.google.com/file/d/1PuvBzd … sp=sharing linux-5.15.1.arch1-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K02MSo … sp=sharing linux-headers-5.15.1.arch1-1-x86_64.pkg.tar.zst

@loqs thanks. I confirm that booting this kernel with laptop lid closed (using my external monitor) fixes the 5.15.1 regression and behaves like 5.14.16 did, as stated in https://bugs.archlinux.org/task/72658#comment203488  / section "==== 2. Stable kernel 5.14.16, laptop lid closed ====" : with this kernel, I *do* see the LUKS prompt, albeit graphically glitched as shown in picture https://bugs.archlinux.org/task/72658?getfile=20839 .

Any particular likely culprits among the diff you shared above? How do you suggest to bisect this?

Finally, do you have extra ideas about the initial reason for the graphical glitching? (Which started happening a few months ago, maybe around 5.12 / 5.13 / 5.14). Does it feel like a bug, or are there kernel boot flags we could try? Any extra test I could run? (Aside the nomodeset i915.modeset=0 flags I already tried).

Last edited by ronjouch (2021-11-08 13:08:11)

Al.Piotrowicz

Member

Registered: 2017-08-07

Posts: 128

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

After compiling kernel within the most recent changes from trunk - commit d20628466c3553fd52d18fa80b6c3126de85236c (reverted by loqs in this topic), I have the same "loading initrd" stuck like the OP, but I'm still able to type my luks password despite the lack of prompt console. System still boots fine, but the main machine screen stills stuck. Thus I'm able to use it by the secondary ssh login.

***EDIT***

According to experienced problem the solution is in provided bug report by OP https://bugs.archlinux.org/task/72645

For myself, the missing config options were:

CONFIG_FB_UVESA=m
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

Last edited by Al.Piotrowicz (2021-11-08 18:39:55)

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

Re: [SOLVED][Testing] Kernel 5.15.1 stuck at "Loading initial ramdisk ..."

I have the same "loading initrd" stuck like the OP, but I'm still able to type my luks password despite the lack of prompt console. System still boots fine, but the main machine screen stills stuck.

Just to precise / re-qualify your mid-sentence "like the OP, but ...", I have the exact same experience, no difference: system appears frozen, but I'm still able to type my luks password despite the lack of prompt console.

According to experienced problem the solution is in provided bug report by OP https://bugs.archlinux.org/task/72645

For myself, the missing config options were:

CONFIG_FB_UVESA=m
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

I confirm these flags produces for me too (Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07), Kernel driver in use: i915) a working LUKS prompt. One that is graphically glitched, but that's another pre-existing bug unrelated to 5.15.

Thanks everyone . Marking this thread again as [SOLVED], closing my Flyspray bug, and confirming in the other one that these flags work for me.