Arch Linux

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Boot hangup on "Loading initial ramdisk" after update & restic backup

Hey Archers!

I am running an Arch64 server with the hardened kernel and update on a daily basis.

> uname -a 
Linux archserver 6.14.9-hardened1-1-hardened #1 SMP PREEMPT_DYNAMIC Mon, 02 Jun 2025 20:56:08 +0000 x86_64 GNU/Linux

Everything worked fine for many months. Yesterday I did a remote restic backup to the server via sFTP. This also worked fine for the last 141 snapshots.
But this time it caused some hangup and I had to cold reboot.

Since then the boot process hangs at

“Loading initial ramdisk.”

The snapshots are stored on a HDD which is only mounted when I start the backup. So it is fully independent from the system root and boot directory which resides on a seperate SSD.

What I did so far (w/ checking the boot behaviour after every approach and no effect so far…):

(1) Updated (from <arch-chroot> environment):

> sudo arch-chroot /mnt/root
root> sudo pacman -Syu

(2) Added an <echo>-prompt to the <grub.cfg>:

echo    'Loading Linux linux-hardened ...' 
linux   /boot/vmlinuz-linux-lts root=UUID=fa7a634b-f09f-4798-9eb4-98177ad59d64 rw  loglevel=3 nomodeset 
echo    'Loading initial ramdisk ...' 
initrd  /boot/amd-ucode.img /boot/initramfs-linux-hardened.img
echo “Initramfs loaded.”

This prompt is actually reached every time, so I suppose the image is loaded.
But then the boot process hangs again.

(3) Check UUIDs especially for the root directory in <fstab> (from <arch-chroot> environment):

> cat /etc/fstab 
# Static information about the filesystems. 
# See fstab(5) for details. 

# <file system> <dir> <type> <options> <dump> <pass> 
# /dev/sdb2 
UUID="fa7a634b-f09f-4798-9eb4-98177ad59d64"     /               ext4            rw,relatime     0 1 

# /dev/sdb1 
UUID="15fa3d9c-abc9-4f20-8390-f74281322560"     none            swap            defaults        0 0 

# /dev/sda3  
UUID="18d2b4bc-564a-48be-9128-eff85b1d9135"     /home           ext4            rw,relatime     0 1

Excerpt from <blkid> in <arch-chroot> environment:

> blkid
/dev/sdc2: UUID="fa7a634b-f09f-4798-9eb4-98177ad59d64" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ef918bb9-02" 
/dev/sdc3: UUID="18d2b4bc-564a-48be-9128-eff85b1d9135" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ef918bb9-03" 
/dev/sdc1: UUID="15fa3d9c-abc9-4f20-8390-f74281322560" TYPE="swap" PARTUUID="ef918bb9-01"

The different device paths should not matter here since the drives are defined by their UUID, right?

(4) Booted with all installed kernels and also their fallback versions, i.e.

>ls -l /boot 
total 439500 
-rw-r--r-- 1 root root    153600 Mai 22 20:06 amd-ucode.img 
drwxr-xr-x 6 root root      4096 Jun  4 00:09 grub 
-rw------- 1 root root 119448200 Jun  3 13:02 initramfs-linux-fallback.img 
-rw------- 1 root root 119295638 Jun  3 13:01 initramfs-linux-hardened-fallback.img 
-rw------- 1 root root  15172900 Jun  3 13:00 initramfs-linux-hardened.img 
-rw------- 1 root root  15090497 Jun  3 13:02 initramfs-linux.img 
-rw-r--r-- 1 root root  15462912 Mai 31 11:08 vmlinuz-linux 
-rw-r--r-- 1 root root  15254016 Jun  3 10:22 vmlinuz-linux-hardened 

(5) Booted with & without <amd-microcode.img>:

Therefor I interactively changed the kernel parameters in the GRUB menu (1st entry, hit [e]) from

initrd  /boot/amd-ucode.img /boot/initramfs-linux-hardened.img

to

initrd /boot/initramfs-linux-hardened.img

(5) Installed <linux-lts> kernel from <arch-chroot> environment and booted with it:
   

> sudo arch-chroot /mnt/root

#> pacman -Syu linux-lts

#> grub-mkconfig -o /boot/grub/grub.cfg 
Generating grub configuration file ... 
Found linux image: /boot/vmlinuz-linux-lts 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-lts.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-lts-fallback.img 
Found linux image: /boot/vmlinuz-linux-hardened 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-hardened.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-hardened-fallback.img 
Found linux image: /boot/vmlinuz-linux 
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img 
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-fallback.img 
Warning: os-prober will not be executed to detect other bootable partitions. 
Systems on them will not be added to the GRUB boot configuration. 
Check GRUB_DISABLE_OS_PROBER documentation entry. 
Adding boot menu entry for UEFI Firmware Settings ... 
done

I will go with the LTS kernel in future anyways (for obvious reasons ;> ). However, boot still hangs.

Additional information

See the full <grub.cfg> here [pastebin.com]
Further system specs:
    * CPU: AMD Phenom II X6 1055T
    * Motherboard: ASUS M4A87TD EVO
    * RAM: 2x 2GB DDR3 1333Mhz PC3-10660 CL7-7-7-24
    * GFX: nVidia Geforce GTX 460

Every help on further debugging is much appreciated!

.toadie

Last edited by TinkerToadie (2025-06-04 09:35:31)

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

(6) Tried the following kernel parameters with the LTS kernel

as recommended here in the forums.

iommu=off
amd_iommu=off
acpi=off

Neither one alone nor all in combination did change anything.

EDIT: Maybe the topic should be moved to the "Kernel & Hardware" forum, I just got aware of that. Thank you mods!

Last edited by TinkerToadie (2025-06-04 12:56:41)

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

Nobody?

Every help is highly appriciated, I don't know how to proceed.

Best,
.toadie

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 13,756

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

GFX: nVidia Geforce GTX 460

Are you using nouveau or a nvidia-dkms version from aur ?
If dkms, are the headers for all installed kernels present ?

There is a distinct possibility that the boot does continue but just doesn't show messages .
Boot the chroot environment and look at the journal for the previous boot using journalctl -b -1 .
Use a bigger number for older journals.

Moderator Note
moving to Kernel & Hardware as requested
(next time please use the report button at bottom right to contact mods)

Last edited by Lone_Wolf (2025-06-08 11:33:05)

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

Thank you for moving the thread!

Ok, some more outputs from arch-chroot.

you using nouveau or a nvidia-dkms version from aur ?

I don't remember installing any nVidia driver explicitly. The server is run headless, with the graphics card only attached for occasional debugging. I don't need any accelaration and CLI always worked well out of the box. These both I actually installed after the error occurred:

># pacman -Q | grep -E "nvidia|dkms|nouveau"
nvidia-lts 1:570.153.02-4
nvidia-utils 570.153.02-1

># pacman -Q | grep headers
linux-api-headers 6.14-1
linux-hardened-headers 6.14.9.hardened1-1
linux-headers 6.14.9.arch1-1
linux-lts-headers 6.12.31-1

Boot the chroot environment and look at the journal for the previous boot using journalctl -b -1 .

The last journal entries end with

># journalctl -b -1
...
...
Jun 03 00:45:25 archlab systemd[1]: dev-sdc1.swap: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Deactivated swap /dev/sdc1.
Jun 03 00:45:25 archlab systemd[1]: dev-disk-by\x2duuid-15fa3d9c\x2dabc9\x2d4f20\x2d8390\x2df74281322560.swap: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Deactivated swap /dev/disk/by-uuid/15fa3d9c-abc9-4f20-8390-f74281322560.
Jun 03 00:45:25 archlab systemd[1]: Reached target Unmount All Filesystems.
Jun 03 00:45:25 archlab systemd[1]: Reached target Late Shutdown Services.
Jun 03 00:45:25 archlab systemd[1]: systemd-poweroff.service: Deactivated successfully.
Jun 03 00:45:25 archlab systemd[1]: Finished System Power Off.
Jun 03 00:45:25 archlab systemd[1]: Reached target System Power Off.
Jun 03 00:45:25 archlab systemd[1]: Shutting down.
Jun 03 00:45:25 archlab systemd-shutdown[1]: Syncing filesystems and block devices.
Jun 03 00:45:25 archlab systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jun 03 00:45:25 archlab systemd-journald[295]: Journal stopped
lines 2121-2157/2157 (END)

Looks like normal shudown process to me. But it seems that for some reason none of the following boot attempts was logged!?

.toadie

Last edited by TinkerToadie (2025-06-08 15:03:40)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,734

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

I don't remember installing any nVidia driver explicitly. The server is run headless, with the graphics card only attached for occasional debugging. I don't need any accelaration and CLI always worked well out of the box. These both I actually installed after the error occurred:

># pacman -Q | grep -E "nvidia|dkms|nouveau"
nvidia-lts 1:570.153.02-4
nvidia-utils 570.153.02-1

Those should definitely NOT be installed on your system. They're too new, and nvidia-utils will blacklist nouveau, so nothing will work.

Last edited by Scimmia (2025-06-08 15:15:46)

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

># pacman -Q | grep -E "nvidia|dkms|nouveau"
nvidia-lts 1:570.153.02-4
nvidia-utils 570.153.02-1

Those should definitely NOT be installed on your system. They're too new, and nvidia-utils will blacklist nouveau, so nothing will work.

Removed both packages. No effect.
I also removed the <nomodeset> kernel parameter. No effect.

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 13,756

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

You need to post the full journal as the cause of the issue might be much earlier then the end of the journal.

Assuming you have internet connection from the chroot, use

# journalclt -b -1 | curl -F 'file=@-' 0x0.st

It will upload the full journal to a public hosting site and output a link, post that link.

There should be no confidential data in the journal, but there's one caveat .
If your server is accessible through a public ip-address you should obfuscate (parts of) the address.

For public ipv4 addresses replacing the last 3 numbers is usually good enough, but replacing all 4 is ok also.
example : 118.57.14.190 > 118.w.x.y.z > w.x.y.z

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

TinkerToadie

Member

Registered: 2023-03-17

Posts: 11

Re: Boot hangup on "Loading initial ramdisk" after update & restic backup

I realized that I could not enter my BIOS. Thus, I did a CMOS reset. Bad idea, now exactly <nothing> happens anymore when I try to boot.
This is no longer an archLinux problem but probably some hardware defect (and I am fucked...)

The upload to 0x0.st still is a nice feature, gonna use it in future

Thanks for your patience, guys.

Best
.toadie