[SOLVED] initramfs unpacking failed invalid magic

shoedrip · 2024-01-18 17:34:15

Hello there,

Never had an issue after 10+ years of using ArchLinux. But this issue stomped me.
My system refuses to boot past the systemd-boot screen, I get a kernel panic with "initramfs unpacking failed invalid magic at start of compressed archive"

I initially thought this was because my boot partition was full for some reason, and because of that the upgrade hook that runs mkinitcpio failed because and I didn't notice, so I thought, no biggie, I'll just run a arch install usb, chroot into my system, and rebuild the initramfs after looking up the real issue.
Well, the same thing happened when booting from a live usb. The image was from last year so I thought, maybe I screwed dd'ing the image, I'll just do it again with the latest install iso. Nope, same thing. Tried to boot the iPXE from UEFI shell and it just hangs after the init message, might be because it's a pretty old motherboard...
I started thinking it might be my ram became bad. Ran memtest from the install iso, completed a whole pass, no error.
The system is a fairly old sandy bridge, so then I thought maybe it's a bad microcode, but that wouldn't explain why the old install iso failed to boot too, and issues with it shouldn't persist between boots.
Last resort, I downloaded the latest ubuntu install iso, and this one actually booted. I rebuilt my initramfs, but it didn't change anything. Also tried to switch from zstd to gzip, nothing.
I can't think of anything that changed significantly besides updating my system, but I don't think anything in that update that could have broke my setup up that badly, especially considering the live usbs themselves also kernel panic past the bootloader.

Any clue on what to look for next? What would make the ubuntu live iso boot, but not arch? Is the initramfs setup differently?
I could try to start from an uncompressed initramfs, though my boot partition is too small at the moment (but it's 50% used when compressing the initramfs) I'd need to do some reallocations first...

I'd appreciate any help, thanks for reading.

Edit: was a kernel issue. Downgraded to a very old 5.15 kernel, will bisect to the last good kernel later.

Last edited by shoedrip (2024-01-18 18:52:16)

V1del · 2024-01-18 17:47:31

Sounds similar to https://bbs.archlinux.org/viewtopic.php?id=290093 -- and potentially some change in the kernel's own zstd or so compression algos. Maybe try the same workaround of setting maxcpus=1 on the kernel commandline to see whether you get past this.

FWIW I'd also not rule out filesystem/partition errors what filesystem is this? FAT32 for an EFI? I personally don't trust those and had similar issues on shutdown when having those, maybe run a fsck/check your disk health by running a SMART test and then checking/posting

smartctl -a

shoedrip · 2024-01-18 17:56:31

Tried maxcpus=1 and no changes.
I doubt it's a recent change since even the november 2023 install iso didn't work either, though maybe I should look for something even older first before going to this conclusion, as the ubuntu iso seems to be way older (22.04.3).
I thought about disk integrity, but it's unlikely that it affect both my SSD and my USB drive in the exact same way.
However the thread you linked raises a good point about the CPU possibly being bad. Though I never did any overclocking (it doesn't even support it), but it would be nice if there was some kind of "cputest" program.
Thanks for the suggestions!

EDIT: PROBLEM SOLVED, managed to boot a arch install iso from december 2021, so this cements it as a kernel issue, thanks!

Last edited by shoedrip (2024-01-18 18:09:32)

seth · 2024-01-18 18:12:09

I'd rather say it's because that kernel used a different compression algorithm…

V1del · 2024-01-18 20:51:48

This also shouldn't "generally" happen... is your CPU stable when doing e.g. a stress test?

shoedrip · 2024-01-19 07:03:57

Never did a stress test, nor had any stability issue in the multiple years I've had this CPU. Finding the kernel version for which it stops booting could shed some light on what's happening, if it's a real kernel bug or a newer kernel feature exposing a CPU issue... I'll report back with results

seth · 2024-01-19 09:05:11

The issue seems to have started October, but the kernel uses zstd since much longer - and actually before december 2021, https://archlinux.org/news/moving-to-zs … kinitcpio/

shoedrip · 2024-01-19 11:25:11

During bisecting I came across a ton of kernel versions that failed to boot for various reasons, but eventually started from a much narrower period and figured out the issue is much more recent.
6.6.10-arch1-1 is the last working kernel for me, linux-6.7.arch1-1 is the first that breaks for me

I guess I'll just switch to LTS in the meanwhile.

Last edited by shoedrip (2024-01-19 11:53:37)

V1del · 2024-01-19 12:28:19

Did you rule out that you're simply running out of space on your /boot partition due to increased space requirements of the initramfs due to the kms hook in mkinitcpio? https://bbs.archlinux.org/viewtopic.php?id=291900

Last edited by V1del (2024-01-19 13:00:18)

shoedrip · 2024-01-19 13:23:03

Yes, my boot partition is only 50% used, even with both the 6.7 initramfs and the LTS initramfs.
I just realized the version I singled out doesn't make any sense now, since the kernels used on the live images worked when running directly on my machine, but failed when booting directly from the usb drive.
Anyway this is getting too complicated. Switching to LTS fixed my issue, I'll stay there until something else breaks.

seth · 2024-01-19 15:01:48

The main kernel will eventually become the next LTS, so you want to keep an eye on this and if it doesn't magically fix, look into the cause.
Otherwise you'll end up in calamities (you need to update but can't)

shoedrip · 2024-01-19 18:02:13

My intuition would be to build linux from the git source and bissect until I find the offending commit, except I don't understand how the kernel is tagged/being developed, the code for 6.6.12 isn't on the same branch as 6.7 so I don't know how to bissect between these two.
Is it, like, 6.7 is based on 6.6 but adds new features while also merging fixes from 6.6.X that are being worked on in parallel?

seth · 2024-01-19 22:22:17

https://wiki.archlinux.org/title/Bisect … s_with_Git
Every version that has existed in the main (not! the LTS) kernel is reachable this way.

bamboo1539 · 2024-05-12 21:34:07

I encountered this error message but it does not panic and proceeds to boot as normal. However I was annoyed by it and I was able to make it go away. Of course I do not understand why. If anyone knows better, please chime in.

I'm going to describe what I did. First, the common action path, then I will describe the bad and the good.

The Common Action Path

I have an AMD Ryzen 5 2600, so I installed amd-ucode. For reference, I followed the instructions here https://wiki.archlinux.org/title/microcode#mkinitcpio

Specifically, this is what I did. I added microcode to the HOOKS:

Note the order of autodetect and microcode

# /etc/mkinitcpio.conf
...
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck numlock)
...

Then rebuilt the image

$ sudo mkinitcpio -P

I make sure I have the same result as that described by the linked wiki page above:

$ sudo lsinitcpio --early /boot/initramfs-linux.img | less
early_cpio
kernel/
kernel/x86/
kernel/x86/microcode/
kernel/x86/microcode/AuthenticAMD.bin
lib
lib64
usr/
usr/lib/
...

Now we branch out:

The Bad

In my bootloader, I have the following kernel options. With initrd=initramfs-linux.img, I get the "initramfs unpacking failed invalid magic" error message. (But continues to boot normally).

"root=UUID=xxxx resume=UUID=xxxx initrd=initramfs-linux.img loader=vmlinuz-linux rw quiet"

The Good

With initrd=amd-ucode.img, I can make the boot error message go away.

"root=UUID=xxxx resume=UUID=xxxx initrd=amd-ucode.img loader=vmlinuz-linux rw quiet"

The full boot stanza I'm using in rEFInd is the following:

# /boot/EFI/refind/refind.conf
...
menuentry "Arch Linux" {
    volume <PARTUUID of the EFI partition>
    loader vmlinuz-linux
    initrd initramfs-linux.img
    options "root=UUID=<UUID of the root partition> resume=UUID=<UUID of the swap partition> initrd=amd-ucode.img loader=vmlinuz-linux rw quiet"
    submenuentry "Use fallback initramfs" {
        initrd initramfs-linux-fallback.img
    }
    submenuentry "Boot to terminal" {
        add_options "systemd.unit=multi-user.target"
    }
}
...

I hope this helps with getting to the root cause!

seth · 2024-05-13 05:57:45

So you did https://wiki.archlinux.org/title/Microc … ly_loading - specifically https://wiki.archlinux.org/title/Microcode#rEFInd
Not sure how

I encountered this error message but … proceeds to boot as normal.

can have worked but unless this is a fluke or simply caused by the image re-generation, your CPU needs the µcode fix to be able to decompress the initramfs.

Arch Linux

#1 2024-01-18 17:34:15