Boot hangup on "Loading initial ramdisk" after update & restic backup

TinkerToadie · 2025-06-04 09:12:54

Hey Archers!

I am running an Arch64 server with the hardened kernel and update on a daily basis.

> uname -a 
Linux archserver 6.14.9-hardened1-1-hardened #1 SMP PREEMPT_DYNAMIC Mon, 02 Jun 2025 20:56:08 +0000 x86_64 GNU/Linux

Everything worked fine for many months. Yesterday I did a remote restic backup to the server via sFTP. This also worked fine for the last 141 snapshots.
But this time it caused some hangup and I had to cold reboot.

Since then the boot process hangs at

“Loading initial ramdisk.”

The snapshots are stored on a HDD which is only mounted when I start the backup. So it is fully independent from the system root and boot directory which resides on a seperate SSD.

What I did so far (w/ checking the boot behaviour after every approach and no effect so far…):

(1) Updated (from <arch-chroot> environment):

> sudo arch-chroot /mnt/root
root> sudo pacman -Syu

(2) Added an <echo>-prompt to the <grub.cfg>:

echo    'Loading Linux linux-hardened ...' 
linux   /boot/vmlinuz-linux-lts root=UUID=fa7a634b-f09f-4798-9eb4-98177ad59d64 rw  loglevel=3 nomodeset 
echo    'Loading initial ramdisk ...' 
initrd  /boot/amd-ucode.img /boot/initramfs-linux-hardened.img
echo “Initramfs loaded.”

This prompt is actually reached every time, so I suppose the image is loaded.
But then the boot process hangs again.

(3) Check UUIDs especially for the root directory in <fstab> (from <arch-chroot> environment):

> cat /etc/fstab 
# Static information about the filesystems. 
# See fstab(5) for details. 

# <file system> <dir> <type> <options> <dump> <pass> 
# /dev/sdb2 
UUID="fa7a634b-f09f-4798-9eb4-98177ad59d64"     /               ext4            rw,relatime     0 1 

# /dev/sdb1 
UUID="15fa3d9c-abc9-4f20-8390-f74281322560"     none            swap            defaults        0 0 

# /dev/sda3  
UUID="18d2b4bc-564a-48be-9128-eff85b1d9135"     /home           ext4            rw,relatime     0 1

Excerpt from <blkid> in <arch-chroot> environment:

> blkid
/dev/sdc2: UUID="fa7a634b-f09f-4798-9eb4-98177ad59d64" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ef918bb9-02" 
/dev/sdc3: UUID="18d2b4bc-564a-48be-9128-eff85b1d9135" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ef918bb9-03" 
/dev/sdc1: UUID="15fa3d9c-abc9-4f20-8390-f74281322560" TYPE="swap" PARTUUID="ef918bb9-01"

The different device paths should not matter here since the drives are defined by their UUID, right?

(4) Booted with all installed kernels and also their fallback versions, i.e.

>ls -l /boot 
total 439500 
-rw-r--r-- 1 root root    153600 Mai 22 20:06 amd-ucode.img 
drwxr-xr-x 6 root root      4096 Jun  4 00:09 grub 
-rw------- 1 root root 119448200 Jun  3 13:02 initramfs-linux-fallback.img 
-rw------- 1 root root 119295638 Jun  3 13:01 initramfs-linux-hardened-fallback.img 
-rw------- 1 root root  15172900 Jun  3 13:00 initramfs-linux-hardened.img 
-rw------- 1 root root  15090497 Jun  3 13:02 initramfs-linux.img 
-rw-r--r-- 1 root root  15462912 Mai 31 11:08 vmlinuz-linux 
-rw-r--r-- 1 root root  15254016 Jun  3 10:22 vmlinuz-linux-hardened

(5) Booted with & without <amd-microcode.img>:

Therefor I interactively changed the kernel parameters in the GRUB menu (1st entry, hit [e]) from

initrd  /boot/amd-ucode.img /boot/initramfs-linux-hardened.img

to

initrd /boot/initramfs-linux-hardened.img

(5) Installed <linux-lts> kernel from <arch-chroot> environment and booted with it:

> sudo arch-chroot /mnt/root

#> pacman -Syu linux-lts

#> grub-mkconfig -o /boot/grub/grub.cfg 
Generating grub configuration file ... 
Found linux image: /boot/vmlinuz-linux-lts 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-lts.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-lts-fallback.img 
Found linux image: /boot/vmlinuz-linux-hardened 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-hardened.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-hardened-fallback.img 
Found linux image: /boot/vmlinuz-linux 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-fallback.img 
Warning: os-prober will not be executed to detect other bootable partitions. 
Systems on them will not be added to the GRUB boot configuration. 
Check GRUB_DISABLE_OS_PROBER documentation entry. 
Adding boot menu entry for UEFI Firmware Settings ... 
done

I will go with the LTS kernel in future anyways (for obvious reasons ;> ). However, boot still hangs.

Additional information

See the full <grub.cfg> here [pastebin.com]
Further system specs:
* CPU: AMD Phenom II X6 1055T
* Motherboard: ASUS M4A87TD EVO
* RAM: 2x 2GB DDR3 1333Mhz PC3-10660 CL7-7-7-24
* GFX: nVidia Geforce GTX 460

Every help on further debugging is much appreciated!

.toadie

Last edited by TinkerToadie (2025-06-04 09:35:31)

TinkerToadie · 2025-06-04 12:53:39

(6) Tried the following kernel parameters with the LTS kernel

as recommended here in the forums.

iommu=off
amd_iommu=off
acpi=off

Neither one alone nor all in combination did change anything.

EDIT: Maybe the topic should be moved to the "Kernel & Hardware" forum, I just got aware of that. Thank you mods!

Last edited by TinkerToadie (2025-06-04 12:56:41)

TinkerToadie · Today 10:08:26

Nobody?

Every help is highly appriciated, I don't know how to proceed.

Best,
.toadie

Lone_Wolf · Today 11:20:18

GFX: nVidia Geforce GTX 460

Are you using nouveau or a nvidia-dkms version from aur ?
If dkms, are the headers for all installed kernels present ?

There is a distinct possibility that the boot does continue but just doesn't show messages .
Boot the chroot environment and look at the journal for the previous boot using journalctl -b -1 .
Use a bigger number for older journals.

Moderator Note
moving to Kernel & Hardware as requested
(next time please use the report button at bottom right to contact mods)

Last edited by Lone_Wolf (Today 11:33:05)

TinkerToadie · Today 15:03:11

Thank you for moving the thread!

Ok, some more outputs from arch-chroot.

Lone_Wolf wrote:

you using nouveau or a nvidia-dkms version from aur ?

I don't remember installing any nVidia driver explicitly. The server is run headless, with the graphics card only attached for occasional debugging. I don't need any accelaration and CLI always worked well out of the box. These both I actually installed after the error occurred:

># pacman -Q | grep -E "nvidia|dkms|nouveau"
nvidia-lts 1:570.153.02-4
nvidia-utils 570.153.02-1

># pacman -Q | grep headers
linux-api-headers 6.14-1
linux-hardened-headers 6.14.9.hardened1-1
linux-headers 6.14.9.arch1-1
linux-lts-headers 6.12.31-1

Lone_Wolf wrote:

Boot the chroot environment and look at the journal for the previous boot using journalctl -b -1 .

The last journal entries end with

># journalctl -b -1
...
...
Jun 03 00:45:25 archlab systemd[1]: dev-sdc1.swap: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Deactivated swap /dev/sdc1.
Jun 03 00:45:25 archlab systemd[1]: dev-disk-by\x2duuid-15fa3d9c\x2dabc9\x2d4f20\x2d8390\x2df74281322560.swap: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Deactivated swap /dev/disk/by-uuid/15fa3d9c-abc9-4f20-8390-f74281322560.
Jun 03 00:45:25 archlab systemd[1]: Reached target Unmount All Filesystems.
Jun 03 00:45:25 archlab systemd[1]: Reached target Late Shutdown Services.
Jun 03 00:45:25 archlab systemd[1]: systemd-poweroff.service: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Finished System Power Off.
Jun 03 00:45:25 archlab systemd[1]: Reached target System Power Off.
Jun 03 00:45:25 archlab systemd[1]: Shutting down.
Jun 03 00:45:25 archlab systemd-shutdown[1]: Syncing filesystems and block devices.
Jun 03 00:45:25 archlab systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jun 03 00:45:25 archlab systemd-journald[295]: Journal stopped
lines 2121-2157/2157 (END)

Looks like normal shudown process to me. But it seems that for some reason none of the following boot attempts was logged!?

.toadie

Last edited by TinkerToadie (Today 15:03:40)

Scimmia · Today 15:08:34

TinkerToadie wrote:

I don't remember installing any nVidia driver explicitly. The server is run headless, with the graphics card only attached for occasional debugging. I don't need any accelaration and CLI always worked well out of the box. These both I actually installed after the error occurred:
># pacman -Q | grep -E "nvidia|dkms|nouveau"
nvidia-lts 1:570.153.02-4
nvidia-utils 570.153.02-1

Those should definitely NOT be installed on your system. They're too new, and nvidia-utils will blacklist nouveau, so nothing will work.

Last edited by Scimmia (Today 15:15:46)

Arch Linux

#1 2025-06-04 09:12:54

Boot hangup on "Loading initial ramdisk" after update & restic backup

#2 2025-06-04 12:53:39

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

#3 Today 10:08:26

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

#4 Today 11:20:18

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

#5 Today 15:03:11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

#6 Today 15:08:34

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

Board footer