[SOLVED] Invalid magic number on boot

bysom · 2021-06-02 18:43:41

Hi,

after a system upgrade including linux and linux-headers my system stopped working.

Loading Linux core repo kernel...
error: invalid magic number.
Loading initial ramdisk...
error: you need to load the kernel first

I've already tried to reinstall linux and linux-headers, tried `mkinitcpio -P`, to rebuild my grub config and to reinstall grub. Everything without error messages.

Some weeks ago I had the same issue, but here everything was fine again after I reinstalled linux and linux-headers.

I've used the grub command line. I'm not sure if I used it correctly or if it is a hint:

grub> ls (hd1,gpt3)
    Partition hd1,gpt3: No known filesystem detected - Partition start at 37315584KiB - Total size 3869702983.5KiB

It is a btrfs filesystem and it works.

I hope someone has an idea.

EDIT: My solution was to upgrade grub to 2.06 RC1. Details are in the thread.

Last edited by bysom (2021-06-07 19:52:23)

seth · 2021-06-02 20:06:13

Suggests /boot/vmlinuz-linux is corrupt.
Is the boot partition full (no space left)?
What dow "file /boot/vmlinuz-linux" say and what is the md5 of the file?
If thereÄs enough space, can you isntall and boot the lts kernel?

bysom · 2021-06-03 09:26:48

Hi, thanks for your answer.

There is enough space, since i have no extra /boot partition. I've encrypted my entire system including /boot. Therefore I have an extra efi partition with grub, that decrypts my btrfs and then boots the kernel. Mostly like it is described in the Wiki

# df -h
...
/dev/mapper/cryptroot        3,7T    2,3T  1,4T   64% /mnt
/dev/mapper/cryptroot        3,7T    2,3T  1,4T   64% /mnt/home
/dev/mapper/cryptroot        3,7T    2,3T  1,4T   64% /mnt/.snapshots
/dev/sda1                    599M    352K  599M    1% /mnt/efi

I've also installed the lts kernel. Both aren't working.

# file /mnt/boot/vmlinuz-linux
/mnt/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.12.8-arch1-1 (linux@archlinux) #1 SMP PREEMPT Fri, 28 May 2021 15:10:20 +0000, RO-rootFS, swap_dev 0x8, Normal VGA

# file /mnt/boot/vmlinuz-linux-lts 
/mnt/boot/vmlinuz-linux-lts: Linux kernel x86 boot executable bzImage, version 5.10.41-1-lts (linux-lts@archlinux) #1 SMP Fri, 28 May 2021 15:03:28 +0000, RO-rootFS, swap_dev 0x8, Normal VGA

Here the md5 sums

# md5sum /mnt/boot/vmlinuz-linux
04da1eb5b1d97e228936fbb613277406  /mnt/boot/vmlinuz-linux

# md5sum /mnt/boot/vmlinuz-linux-lts 
f525237336ec905a8af0002ac457d1dc  /mnt/boot/vmlinuz-linux-lts

I've also tried a `btrfs check` without errors.

Thanks for your help!

seth · 2021-06-03 13:42:56

It would seem that either the decryption fails or maybe is just too slow
Can you "ls" the partition after "insmod btrfs"?

bysom · 2021-06-03 17:30:34

Okay, I have some results.

After the cryptomount command:

> ls (crypto0)
Device crypto0: Filesystem type btrfs - Label `cepa`, UUID <the uuid> - Sector size 512B - Total size 3869700935.5KiB

I have tried a `ls /@/boot` after the `echo 'Loading Linux linux-lts ...'` and it lists the files in my /boot directory. After that I have tried a `file --is-x86-linux /@/boot/vmlinuz-linux` and got no result (and also no error). Is there something else I should try?

bysom · 2021-06-03 18:37:26

I've tried a `sleep 30` before loading the kernel, which hasn't changed anything.

After that I have chrooted my arch again and reinstalled linux, linux-lts, and the headers and now something changed. If I try to boot the LTS kernel it passes the 'Loading Linux linux-lts ...' section and then I get a kernel panic in the following 'Loading initial ramdisk ...' section. The error message before the kernel panic is

Initramfs unpacking failed: invalid magic at start of compressed

The normal linux kernel still fails at the first section 'Loading Linux linux ...'

Last edited by bysom (2021-06-03 18:38:35)

seth · 2021-06-04 05:19:17

Re-cap:
This is a regression, the config used to work befoer?
The files end on the encrypted partition intact, ie. they're perfectly accessible offline?
The error has a degree of randomness, eg. vmlinuz-linux-lts (now) loads, but the other files have bad header bytes?

What else was included in the breaking udpate? /var/log/pacman.log?

file --is-x86-linux /@/boot/vmlinuz-linux

returns a boolean.

if file --is-x86-linux /@/boot/vmlinuz-linux; then echo 'kernel is good'; fi

bysom · 2021-06-04 14:48:33

Yes, your recap sounds correct. The config worked before, I can access my files and the error seems to have some kind of randomness.

Here is the pacman log from the last update before the crash: http://ix.io/3oJJ

seth · 2021-06-04 14:55:54

cryptsetup sticks out…
Downgrade that, re-install the kernels and see what happens.

BE PREPARED TO LOSE ACCESS TO THE … oh, wait: you can't boot and will have to chroot anyway

bysom · 2021-06-04 17:48:05

I've downgraded to cryptsetup 2.3.4 and the error is still the same.

Is this setup with an encrypted boot partition that rare, that it could be a bug in grub or somewhere else? The idea to reinstall my arch with a classical encryption setup gets more and more attractive for me. Unless you have still some good ideas. I'm still very happy with your help

seth · 2021-06-04 20:40:21

Did the kernel re-installation change anything about the randomness aspect of the kernel/initramfs "corruption"?
And otherwise, how the status is at the grub shell (ie. testing the file types there) in case this is -somehow- a time/race condition…

bysom · 2021-06-05 20:23:24

Hi, I have some news.

To your questions: The kernel reinstallation made that the error occours at the 'Loading Linux linux-lts ...' again for the lts kernel which made it to the initramfs loading. As far as I remember this state changed with the reinstallation of the kernel.

I have tried your 'if file xy then' example and it obviously returned a false value. I've got a 'kernel is bad' because I've added an else case. It should have been true, right?

After that I saw, that I can use md5sum with grub and checked the md5 sum of the kernel and got a different number than I get from my live system.

From grub

> md5sum /@/boot/vmlinuz-linux
546c...145ce /@/boot/vmlinuz-linux

From a live linux

# md5sum /mnt/boot/vmlinuz-linux
8ed348f0125772a6ef0756b818199fc6  /mnt/boot/vmlinuz-linux

I have asked someone else with an arch linux for a hash sum as a reference and got the 8ed348f0125772a6ef0756b818199fc6 for the actual kernel.

(I've updated the kernel yesterday)

Then I've used cat with grub to view some text files which all looked normal to me. After that I've used md5sum on one of these files and got the same hash numbers with grub and with my live stick.

Last edited by bysom (2021-06-05 21:33:05)

seth · 2021-06-05 22:33:50

If you change the mkinitcpio compression (in doubt to "cat"), do the md5sums for those turn consistent?
https://wiki.archlinux.org/title/Mkinitcpio#COMPRESSION

bysom · 2021-06-06 12:13:21

Hi, I have changed that compression to cat and reinstalled the kernels.

The md5 sums from my live linux (ubuntu 20.04, Kernel 5.8.0, cryptsetup 2.2.2, btrfs-progs v5.4.1). Those from my chrooted arch are the same.

f14f10d20670a7638af02f4ae11b8c2e  /mnt/boot/initramfs-linux.img
2cda8759540127bf587869679441188b  /mnt/boot/initramfs-linux-lts.img
8ed348f0125772a6ef0756b818199fc6  /mnt/boot/vmlinuz-linux
afe5da8d09ed00335cea004e1ed79f65  /mnt/boot/vmlinuz-linux-lts

The md5sums from grub

f14f10d20670a7638af02f4ae11b8c2e  /@/boot/initramfs-linux.img
2cda8759540127bf587869679441188b  /@/boot/initramfs-linux-lts.img
e36e627...3a053  /@/boot/vmlinuz-linux
7727430...f441de  /@/boot/vmlinuz-linux-lts

That ones from the initramfs are correct. And that one from the linux is different now.

Last edited by bysom (2021-06-06 12:18:59)

seth · 2021-06-06 12:34:36

Smells as if the grub btrfs module doesn't like compressed files (at least not zstd)?
If you check a random zstd compressed file from the grub shell, does its md5 deviate?
Or do you maybe use https://wiki.archlinux.org/title/Btrfs#Compression ?

bysom · 2021-06-06 21:57:33

Yes, I'm using btrfs compression with zstd.

I've checked the results with a normal file

Linux point of view

c3b6db02c809b04751fa5c7c0f430df4  /mnt/root/Testfile.mp4
0a02ed72843d8893bd5616cf58dd8737  /mnt/root/Testfile.mp4.zst

grub point of view

41bf8909...cdc8da  /@/root/Testfile.mp4
3283ab1ac...281f11e  /@/root/Testfile.mp4.zst

Both files are different here. There could be some randomness, I think.

Last edited by bysom (2021-06-06 22:27:06)

seth · 2021-06-06 22:03:11

What if you "chattr -c /boot" and update the kernel?

Edit: also notice the warning and to have grub-install post 2019-07-05

Last edited by seth (2021-06-06 22:06:14)

bysom · 2021-06-07 19:45:18

Hi, I have tried that chattr -c /boot thing and it hasn't changed anything. After that I have checked the md5sums with grub again and got for one file an error message containing 'error: zstd data corrupted.'. After that I have repeated that btrfs check just in case without errors.

Since the error only occurred with grub and there is already a rc1 for the next grub release, which seems to be really rare, I thought that I could try grub-git from the aur (what if it is a grub bug and no one fixes it before the release, was my first thought). So I installed grub-git, then executed grub-install, restored the grub config and got a booting system. I'm a bit happy now. And I am very thankful for your help and your patience, seth! A very positive experience. Let me know if I can do something in reward.

seth · 2021-06-07 19:47:19

Let me know if I can do something in reward.

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

;-)

bysom · 2021-06-07 19:49:57

Of course

Last edited by bysom (2021-06-07 19:54:38)

Arch Linux

#1 2021-06-02 18:43:41

[SOLVED] Invalid magic number on boot

#2 2021-06-02 20:06:13

Re: [SOLVED] Invalid magic number on boot

#3 2021-06-03 09:26:48

Re: [SOLVED] Invalid magic number on boot

#4 2021-06-03 13:42:56

Re: [SOLVED] Invalid magic number on boot

#5 2021-06-03 17:30:34

Re: [SOLVED] Invalid magic number on boot

#6 2021-06-03 18:37:26

Re: [SOLVED] Invalid magic number on boot

#7 2021-06-04 05:19:17

Re: [SOLVED] Invalid magic number on boot

#8 2021-06-04 14:48:33

Re: [SOLVED] Invalid magic number on boot

#9 2021-06-04 14:55:54

Re: [SOLVED] Invalid magic number on boot

#10 2021-06-04 17:48:05

Re: [SOLVED] Invalid magic number on boot

#11 2021-06-04 20:40:21

Re: [SOLVED] Invalid magic number on boot

#12 2021-06-05 20:23:24

Re: [SOLVED] Invalid magic number on boot

#13 2021-06-05 22:33:50

Re: [SOLVED] Invalid magic number on boot

#14 2021-06-06 12:13:21

Re: [SOLVED] Invalid magic number on boot

#15 2021-06-06 12:34:36

Re: [SOLVED] Invalid magic number on boot

#16 2021-06-06 21:57:33

Re: [SOLVED] Invalid magic number on boot

#17 2021-06-06 22:03:11

Re: [SOLVED] Invalid magic number on boot

#18 2021-06-07 19:45:18

Re: [SOLVED] Invalid magic number on boot

#19 2021-06-07 19:47:19

Re: [SOLVED] Invalid magic number on boot

#20 2021-06-07 19:49:57

Re: [SOLVED] Invalid magic number on boot

Board footer