You are not logged in.

Pages: 1

#1 2022-12-20 12:29:15

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Broken initramfs with compression cat

I have a system, that did not boot with my backup image resulting in this error.
At first I thought there was an error with systemd due to the stacktrace, but the first line of the error was actually what lead me on the right track.
If I compress the initramfs image with any algorithm other than "cat" (which is my default), the system boots fine.
I can reproducibly trigger the error by setting COMPRESSION=cat in /etc/mkinitcpio.conf and rebuilding the initramfs.
My question is: Why? I have literally hundreds of systems set up with COMPRESSION=cat and never encountered this issue.

I also tried to move the EFI partition on the disk as I also suspected a broken NVME drive at first.
But the error can be reproduced with the EFI partition moved to the other end of the disk as well.

Some details about the system in question:
"terra" branded Intel NUC
Journal from a successful boot: http://ix.io/4j7q

$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 142
model name	: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz
stepping	: 9
microcode	: 0xf0
cpu MHz		: 2400.000
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple pml ept_mode_based_exec
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_stale_data retbleed
bogomips	: 4800.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 142
model name	: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz
stepping	: 9
microcode	: 0xf0
cpu MHz		: 1383.996
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple pml ept_mode_based_exec
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_stale_data retbleed
bogomips	: 4800.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 142
model name	: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz
stepping	: 9
microcode	: 0xf0
cpu MHz		: 2400.000
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple pml ept_mode_based_exec
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_stale_data retbleed
bogomips	: 4800.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 142
model name	: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz
stepping	: 9
microcode	: 0xf0
cpu MHz		: 2400.000
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple pml ept_mode_based_exec
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_stale_data retbleed
bogomips	: 4800.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 119,2G  0 disk 
├─nvme0n1p1 259:1    0   500M  0 part /boot
└─nvme0n1p2 259:2    0 118,8G  0 part /
$ lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5904] (rev 02)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: skl_uncore
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 620 [8086:5916] (rev 02)
	DeviceName: Onboard - Video
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: i915
	Kernel modules: i915
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:9d2f] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:9d31] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: intel_pch_thermal
	Kernel modules: intel_pch_thermal
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-LP CSME HECI #1 [8086:9d3a] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] [8086:9d03] (rev 21)
	DeviceName: Onboard - SATA
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #1 [8086:9d10] (rev f1)
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 [8086:9d15] (rev f1)
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: pcieport
00:1c.7 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #8 [8086:9d17] (rev f1)
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: pcieport
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #9 [8086:9d18] (rev f1)
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: pcieport
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point LPC Controller/eSPI Controller [8086:9d4e] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
	DeviceName: Onboard - Sound
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel, snd_soc_skl
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-V [8086:15d8] (rev 21)
	DeviceName: Onboard - Ethernet
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: e1000e
	Kernel modules: e1000e
3a:00.0 Network controller [0280]: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78)
	Subsystem: Intel Corporation Device [8086:9010]
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi
3b:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5229 PCI Express Card Reader [10ec:5229] (rev 01)
	Subsystem: Intel Corporation Device [8086:2068]
	Kernel driver in use: rtsx_pci
	Kernel modules: rtsx_pci
3c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
	Subsystem: Samsung Electronics Co Ltd SM963 2.5" NVMe PCIe SSD [144d:a801]
	Kernel driver in use: nvme
$ cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=('i915' 'radeon')

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base systemd autodetect modconf block filesystems keyboard fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="cat"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()
$ cat /etc/mkinitcpio.d/linux-lts.preset 
# mkinitcpio preset file for the 'linux-lts' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux-lts"

PRESETS=('default')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-linux-lts.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-linux-lts-fallback.img"
fallback_options="-S autodetect"
 $ uname -a
Linux 1871 5.15.84-1-lts #1 SMP Mon, 19 Dec 2022 12:47:08 +0000 x86_64 GNU/Linux
 $ cat /boot/loader/entries/digital-signage.conf 
title	Digital Signage
linux	/vmlinuz-linux-lts
initrd	/amd-ucode.img
initrd	/intel-ucode.img
initrd	/initramfs-linux-lts.img
options	init=/usr/lib/systemd/systemd root=LABEL=root rw intel_idle.max_cstate=1 fsck.repair=yes
$ bootctl list
         type: Boot Loader Specification Type #1 (.conf)
        title: Maintenance
           id: maintenance.conf
       source: /boot/loader/entries/maintenance.conf
        linux: /vmlinuz-linux-lts
       initrd: /amd-ucode.img
               /intel-ucode.img
               /initramfs-linux-lts.img
      options: init=/usr/lib/systemd/systemd root=LABEL=root rw systemd.unit=multi-user.target

         type: Boot Loader Specification Type #1 (.conf)
        title: Configuration Mode
           id: hidsl-cfg.conf
       source: /boot/loader/entries/hidsl-cfg.conf
        linux: /vmlinuz-linux-lts
       initrd: /amd-ucode.img
               /intel-ucode.img
               /initramfs-linux-lts.img
      options: init=/usr/lib/systemd/systemd root=LABEL=root rw systemd.unit=hidslcfg.target

         type: Boot Loader Specification Type #1 (.conf)
        title: Digital Signage (default) (selected)
           id: digital-signage.conf
       source: /boot/loader/entries/digital-signage.conf
        linux: /vmlinuz-linux-lts
       initrd: /amd-ucode.img
               /intel-ucode.img
               /initramfs-linux-lts.img
      options: init=/usr/lib/systemd/systemd root=LABEL=root rw intel_idle.max_cstate=1 fsck.repair=yes

         type: Automatic
        title: Reboot Into Firmware Interface
           id: auto-reboot-to-firmware-setup
       source: /sys/firmware/efi/efivars/LoaderEntries-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f
$ bootctl status 
System:
      Firmware: UEFI 2.70 (American Megatrends 5.12)
 Firmware Arch: x64
   Secure Boot: disabled (disabled)
  TPM2 Support: yes
  Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 252.3-1-arch
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/5fd268f9-7119-476f-8b11-fbfa928ed604
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: yes
 System Token: set
       Exists: yes

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/5fd268f9-7119-476f-8b11-fbfa928ed604)
         File: ├─/EFI/systemd/systemd-bootx64.efi (systemd-boot 252.3-1-arch)
               └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 252.3-1-arch)

No boot loaders listed in EFI Variables.

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/5fd268f9-7119-476f-8b11-fbfa928ed604)

Default Boot Loader Entry:
         type: Boot Loader Specification Type #1 (.conf)
        title: Digital Signage
           id: digital-signage.conf
       source: /boot/loader/entries/digital-signage.conf
        linux: /vmlinuz-linux-lts
       initrd: /amd-ucode.img
               /intel-ucode.img
               /initramfs-linux-lts.img
      options: init=/usr/lib/systemd/systemd root=LABEL=root rw intel_idle.max_cstate=1 fsck.repair=yes

What must I do to use an uncompressed image on this system?

PS: I took the screenshot before the system update. The error can still be reproduced after a full system upgrade.
PPS: Generating a working, compressed image and uncompressing it results in the very same error. Why can't this system load an uncompressed image? It does not make any sense (to me).

Last edited by schard (2022-12-20 13:48:37)

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#2 2022-12-20 15:13:53

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 73,204

Re: Broken initramfs with compression cat

When you compare the leading bytes of that image w/ the ones that do work, is there sth. suspicously looking like junk at the head?
You could also diff lsinitcpio to see whether there're different files in there (from the autodetect hook)

Also, quite obviously, make sure there's enough free space on the boot partition (left) - because that'd be a major difference between cat and the most lousy compression algo…

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#3 2022-12-20 18:33:50

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Re: Broken initramfs with compression cat

seth wrote:

When you compare the leading bytes of that image w/ the ones that do work, is there sth. suspicously looking like junk at the head?

I can do that tomorrow, but I don't see the point to compare the bytes of a raw initcpio image vs. the same image file compressed with an arbitrary compression algorithm.

seth wrote:

You could also diff lsinitcpio to see whether there're different files in there (from the autodetect hook)

I will to that tomorrow, too, but I doubt that there'll be a difference, see my PPS.

seth wrote:

Also, quite obviously, make sure there's enough free space on the boot partition (left) - because that'd be a major difference between cat and the most lousy compression algo…

There is. I only generate one image (no fallback) and the EFI partition on which it is stored is 500MB in size.

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#4 2022-12-20 18:41:24

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

maybe something is wrong with your intel amd ucode img instead?

Offline

#5 2022-12-20 18:51:29

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Re: Broken initramfs with compression cat

Good tip. I'll try to get them out of the equation tomorrow.

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#6 2022-12-20 19:08:21

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

Another thing to try is to pass the kernel/initrd to Qemu/KVM. If it's really something with the file itself or how the kernel handles it, you should get the same error in the VM. (It's fine to have errors regarding missing drives etc. as you're not passing those along)

If it works there, then your file is fine - and the corruption is happening elsewhere (ram, storage, bootloader, filesystem, ...).

Unfortunately the kernel does not print any information - like sizes and checksums of initrd data it received. That would be an easy way to verify if it just got cut off for some reason or similar issues... after all if there's a size limit, you would not be hitting it with the compressed variant.

Offline

#7 2022-12-20 21:05:33

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 73,204

Re: Broken initramfs with compression cat

frostschutz wrote:

maybe something is wrong with your intel amd ucode img instead?

schard wrote:

Generating a working, compressed image and uncompressing it results in the very same error.

I can do that tomorrow, but I don't see the point to compare the bytes of a raw initcpio image vs. the same image file compressed with an arbitrary compression algorithm.

I meant the (uncompressed) ones for machines that do work.
A simple "file /boot/initramfs-linux.img" might also help - the thing is supposed to have an indicating header (and file knows the magic bytes).

Though the strange part is that you should™ end up w/ the very same irritation after decompressing the compressed image - whether it's on the fly or on disk.
I'd still bet on the filesize then, if it's not the partition

frostschutz wrote:

if there's a size limit, you would not be hitting it with the compressed variant

sounds interesting and you could compare the sizes of the images on the working systems w/ this one.

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#8 2022-12-20 21:29:19

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

It could also be something weird, like some reserved memory region that's being used for the initrd data and then corrupted by something else.

The cpio is partially extracted after all, otherwise you would not see the error about /init failing to load systemd libraries. So it might be truncated or corrupted at some point.

You could manually build a minimal initrd that has busybox, /init and a large file with distinct pattern (yes | head -c 100M > yesfile).

And then just run hexdump -C /yesfile in the /init.

If uncorrupted, the output should be a simple

00000000  79 0a 79 0a 79 0a 79 0a  79 0a 79 0a 79 0a 79 0a  |y.y.y.y.y.y.y.y.|
*
06400000

If the file is corrupted or truncated, you might see it that way...

Offline

#9 2022-12-21 09:14:22

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Re: Broken initramfs with compression cat

@seth
Uncompressed:

0 ✓ 1871 ~ $ ls -l /boot/initramfs-linux-lts.img 
-rwxr-xr-x 1 root root 41871360 21. Dez 10:06 /boot/initramfs-linux-lts.img
0 ✓ 1871 ~ $ head -c 64 /boot/initramfs-linux-lts.img | hexdump 
0000000 3730 3730 3130 3030 3030 3030 3130 3030
0000010 3030 3138 3461 3030 3030 3030 3030 3030
0000020 3030 3030 3030 3030 3030 3030 3030 3030
0000030 3030 3030 3030 3030 3030 3030 3230 3030
0000040

gzipped:

0 ✓ 1871 ~ $ head -c 64 /boot/initramfs-linux-lts.img | hexdump 
0000000 8b1f 0008 0000 0000 0300 5cb4 907b 5725
0000010 3f59 20fb 318f dd8f 8125 2424 ef4b 9c44
0000020 b58d daf3 9d9d 9ddd c9b0 364e 331b 8fb8
0000030 3ec9 51f0 ce86 ddb9 dee7 727b 95fb dd3e
0000040
0 ✓ 1871 ~ $ ls -l /boot/initramfs-linux-lts.img 
-rwxr-xr-x 1 root root 16778734 21. Dez 10:05 /boot/initramfs-linux-lts.img

Difference in content (none):

0 ✓ 1871 ~ $ mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 5.15.84-1-lts
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating uncompressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
0 ✓ 1871 ~ $ lsinitcpio /boot/initramfs-linux-lts.img > initramfs.list.uncompressed
0 ✓ 1871 ~ $ vim /etc/mkinitcpio.conf 
0 ✓ 1871 ~ $ mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 5.15.84-1-lts
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
0 ✓ 1871 ~ $ lsinitcpio /boot/initramfs-linux-lts.img > initramfs.list.compressed
0 ✓ 1871 ~ $ diff initramfs.list.*

Regarding free space:

0 ✓ 1871 ~ $ mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 5.15.84-1-lts
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating uncompressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
0 ✓ 1871 ~ $ ls -l /boot/
insgesamt 56932
drwxr-xr-x 5 root root     4096 29. Okt 2021  EFI
-rwxr-xr-x 1 root root    51200 22. Nov 07:05 amd-ucode.img
drwxr-xr-x 2 root root     4096 29. Okt 2021  db568e7c985641e6804af4c1e2c7ba54
-rwxr-xr-x 1 root root 41871360 21. Dez 10:12 initramfs-linux-lts.img
-rwxr-xr-x 1 root root  5678080  8. Nov 20:02 intel-ucode.img
drwxr-xr-x 3 root root     4096 21. Dez 09:29 loader
drwxr-xr-x 2 root root     8192  9. Aug 2021  syslinux
-rwxr-xr-x 1 root root 10668896 20. Dez 13:27 vmlinuz-linux-lts
0 ✓ 1871 ~ $ df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
devtmpfs        4,0M       0  4,0M    0% /dev
tmpfs           1,9G     57M  1,9G    3% /dev/shm
tmpfs           762M    9,0M  753M    2% /run
/dev/nvme0n1p2  117G    4,1G  107G    4% /
tmpfs           1,9G     64K  1,9G    1% /tmp
/dev/nvme0n1p1  500M     59M  441M   12% /boot
tmpfs           381M       0  381M    0% /run/user/0
0 ✓ 1871 ~ $

@frostschutz
Removing the Microcode images from the boot process did not affect the error message.

I tried booting with debug to see what compression the kernel detects as per

0 ✓ neumann@workstation ~/Projekte/linux $ grep -rFn "invalid magic at start of compressed archive"
init/initramfs.c:537:			error("invalid magic at start of compressed archive");

But the plethora of messages makes the message immediately being scrolled off the screen, so I can neither read it, not capture it with my shitty phone camera.
Why did the kernel devs remove the Shift+PageUp scroll option again?

Last edited by schard (2022-12-21 09:47:17)

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#10 2022-12-21 09:41:41

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 73,204

Re: Broken initramfs with compression cat

seth wrote:

I meant the (uncompressed) ones for machines that do work.

Comparing the bytes of a compressed and uncompressed image is obviously pointless.
"3730 3730 3130" is 07 07 01, which is wrong according to https://www.ibm.com/docs/en/zos/2.2.0?t … o-archives and not what I get for "cpio -c"

seth wrote:

A simple "file /boot/initramfs-linux.img" might also help - the thing is supposed to have an indicating header (and file knows the magic bytes).

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#11 2022-12-21 09:47:50

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

It's normal for initramfs cpio (--format=newc)

Offline

#12 2022-12-21 09:51:26

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Re: Broken initramfs with compression cat

My workstation:

255 ✗ neumann@workstation ~ $ cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=(i915)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base systemd autodetect keyboard sd-vconsole modconf block mdadm_udev sd-encrypt filesystems fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
COMPRESSION="cat"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()
0 ✓ neumann@workstation ~ $ head -c 64 | hexdump 
^C
130 ✗ neumann@workstation ~ $ head -c 64 /boot/initramfs-linux.img | hexdump 
0000000 3730 3730 3130 3030 3030 3030 3130 3030
0000010 3030 3138 3461 3030 3030 3030 3030 3030
0000020 3030 3030 3030 3030 3030 3030 3030 3030
0000030 3030 3030 3030 3030 3030 3030 3230 3030
0000040
0 ✓ neumann@workstation ~ $ file /boot/initramfs-linux.img 
/boot/initramfs-linux.img: ASCII cpio archive (SVR4 with no CRC)
0 ✓ neumann@workstation ~ $

So I just followed your lead regarding the file size (notice my passive-aggressive file naming):

130 ✗ 1871 ~ $ head -c 40M /dev/urandom > /etc/random-shit
0 ✓ 1871 ~ $ vim /etc/mkinitcpio.conf 
0 ✓ 1871 ~ $ grep random /etc/mkinitcpio.conf 
FILES=(/etc/random-shit)
0 ✓ 1871 ~ $ mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 5.15.84-1-lts
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
0 ✓ 1871 ~ $ ls -l /boot/
insgesamt 73392
drwxr-xr-x 5 root root     4096 29. Okt 2021  EFI
-rwxr-xr-x 1 root root    51200 22. Nov 07:05 amd-ucode.img
drwxr-xr-x 2 root root     4096 29. Okt 2021  db568e7c985641e6804af4c1e2c7ba54
-rwxr-xr-x 1 root root 58728229 21. Dez 10:58 initramfs-linux-lts.img
-rwxr-xr-x 1 root root  5678080  8. Nov 20:02 intel-ucode.img
drwxr-xr-x 3 root root     4096 21. Dez 10:53 loader
drwxr-xr-x 2 root root     8192  9. Aug 2021  syslinux
-rwxr-xr-x 1 root root 10668896 20. Dez 13:27 vmlinuz-linux-lts
0 ✓ 1871 ~ $ ls -lh /boot/
insgesamt 72M
drwxr-xr-x 5 root root 4,0K 29. Okt 2021  EFI
-rwxr-xr-x 1 root root  50K 22. Nov 07:05 amd-ucode.img
drwxr-xr-x 2 root root 4,0K 29. Okt 2021  db568e7c985641e6804af4c1e2c7ba54
-rwxr-xr-x 1 root root  57M 21. Dez 10:58 initramfs-linux-lts.img
-rwxr-xr-x 1 root root 5,5M  8. Nov 20:02 intel-ucode.img
drwxr-xr-x 3 root root 4,0K 21. Dez 10:53 loader
drwxr-xr-x 2 root root 8,0K  9. Aug 2021  syslinux
-rwxr-xr-x 1 root root  11M 20. Dez 13:27 vmlinuz-linux-lts
0 ✓ 1871 ~ $ systemctl reboot

And bingo: screenshot

Last edited by schard (2022-12-21 10:01:47)

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#13 2022-12-21 10:09:48

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

so it should be hitting some size limit, or its using some memory region it's not supposed to be using, or some other form of corruption by the bootloaders / involved filesystems and storage

also run a memtest, just in case

if there was no size limit and the uncompression continued despite the corruption, you could simply attach each initrd file two or three times. files unpacked twice simply overwrite previously unpacked files. so if the first one was corrupted the second one would fix it. but I guess the whole uncompression is simply aborted when it encounters corruption so that idea won't work at all.

Offline

#14 2022-12-21 10:20:25

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,530
Website

Re: Broken initramfs with compression cat

I did a bit of testing and narrowed the critical threshold down to something between 27M and 37M.
The former works, the latter not.
This is the current top limit I tried, which still boots:

0 ✓ 1871 ~ $ ls -l /boot/initramfs-linux-lts.img 
-rwxr-xr-x 1 root root 27300321 21. Dez 11:16 /boot/initramfs-linux-lts.img

Will run memtest asap.

Last edited by schard (2022-12-21 10:20:57)

Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#15 2022-12-21 10:57:34

frostschutz
Member
Registered: 2013-11-15
Posts: 1,605

Re: Broken initramfs with compression cat

not sure if it could be related to your issue - https://github.com/systemd/systemd/issues/25737 systemd-boot fails after updating systemd to 252.3

Offline

Pages: 1

Board footer

Powered by FluxBB