[SOLVED] initramfs modules vs userspace modules

meowxiik · 2024-12-29 13:19:53

Hey,

So I fucked up and managed to upgrade the Linux modules image without upgrading the kernel. Well chaos ensued and everything broke. I fixed it but in a way that is surprising to me. My kernel was `6.12.6` but my modules were `6.12.7`. My modules were possibly also corrupted because even `modprobe -f` refused to work. Interestingly enough, the boot process got pretty far, it only crashed during systemd startup because it failed to mount /boot, which means initramfs went OK. Now however I was a bit fucked. I had no access to a USB for an ArchISO. I had to do a fully offline fully unassisted fix, using only the broken environment. I had a backup of `6.12.6` Linux package on the filesystem but I couldn't replace the mistakenly newer version of Linux in /boot because /boot can't mount. And I couldn't upgrade the /lib/modules because I had no backup of the 6.12.7 kernel package.

As a last ditch attempt I tried to `break` kernel (by adding the break kernel parameter) so I got into initramfs and SOMEHOW initramfs was able to mount /boot so I was able to replace the broken Linux image. My question is. How???? initramfs and userspace both have the same kernel, right? If the `fat` module was loaded in initramfs why did it decide to unload in userspace? Is the kernel module dependent on the .ko.zst file being present on the filesystem which would mean that the root pivot broke the connection?

Cheers

Last edited by meowxiik (2024-12-29 17:05:41)

nl6720 · 2024-12-29 13:26:28

meowxiik wrote:

If the `fat` module was loaded in initramfs why did it decide to unload in userspace?

What makes you think that the fat module was already loaded?

meowxiik · 2024-12-29 15:47:40

nl670 wrote:

What makes you think that the fat module was already loaded?

Because I was able to mount the /dev/sda1 (boot partition) in initramfs using normal mount command

My drive configuration looks like this, /dev/sda1 is the drive that failed to mount in userland (with error "unkown filesystem vfat") but mounted okay in initramfs

➜  ~ lsblk -f
NAME                    FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                                      
├─sda1                  vfat        FAT32          7E82-5180                                15.8G     1% /boot
├─sda2                  LVM2_member LVM2 001       DftfJj-82W0-ojcp-M8Vr-Zaf1-TZn0-L6peLv                
│ ├─ssdgroup-home       ext4        1.0            d7f71876-b59c-4ce9-a384-51e0e10addcb    139.7G    57% /home
│ └─ssdgroup-overlayfs  ext4        1.0            df79badf-5323-43e8-ab08-652a5837c10d     92.8G     0% /overlayfs/overlay
└─sda3                  LVM2_member LVM2 001       Fdj7PE-f6M3-JWsR-22tX-PrgH-GLUw-e7b8c4                
  ├─ssdgroup-config     ext4        1.0            877366c5-646e-48b2-8d0e-3262513a831e      9.2G     0% /overlayfs/layers/config
  ├─ssdgroup-pkg        ext4        1.0            55abdfd5-8162-42a5-84b3-cacb4832c4ed     24.2G    78% /overlayfs/layers/pkg
  ├─ssdgroup-home       ext4        1.0            d7f71876-b59c-4ce9-a384-51e0e10addcb    139.7G    57% /home
  ├─ssdgroup-docker     ext4        1.0            656afa7b-9865-4bbe-8081-409e1ec8bcf7     56.7G    37% /var/lib/docker
  └─ssdgroup-persistent ext4        1.0            ae0b57de-ac13-4735-afe5-71c5009d1f13     16.6G    10% /var/log/journal
                                                                                                         /persistent

meowxiik · 2024-12-29 15:53:21

My /etc/fstab

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/ssdgroup-pkg
UUID=55abdfd5-8162-42a5-84b3-cacb4832c4ed	/         	ext4      	rw,relatime	0 1

UUID=7E82-5180      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

# UUID=d7f71876-b59c-4ce9-a384-51e0e10addcb
/dev/mapper/ssdgroup-home	/home     	ext4      	rw,relatime	0 2

#/dev/mapper/ssdgroup-lxd	/var/lib/lxd	ext4      	rw,relatime	0 2
/dev/mapper/ssdgroup-docker	/var/lib/docker	ext4		rw,relatime	0 2

tmpfs   /tmp         tmpfs   rw,nodev,nosuid,size=10G          0  0
/overlayfs/layers/pkg/swapfile	none	swap	defaults	0 0

/dev/mapper/ssdgroup-persistent	/persistent	ext4		rw,relatime	0 2

My `/etc/mkinitcpio.conf`

# Comments omitted 
# aufsroot is my custom hook that does transparent layered fs for root. Aufs is a lie, it was aufs originally, it is now done by overlayfs lmao
MODULES=(ext4)
BINARIES=()
FILES=()
HOOKS=(base udev autodetect modconf block lvm2 filesystems aufsroot keyboard fsck)

Scimmia · 2024-12-29 15:57:19

meowxiik wrote:

nl670 wrote:
What makes you think that the fat module was already loaded?
Because I was able to mount the /dev/sda1 (boot partition) in initramfs using normal mount command

At which point it was loaded. The aren't loaded just because they're there.

meowxiik · 2024-12-29 16:54:34

Oh so you're saying that had I loaded the module during initramfs the mounting during user-space would've succeeded but because userspace didn't have access to the correct modules it crashed. Yeah that makes sense, thanks!

dimich · 2024-12-30 00:28:47

meowxiik wrote:

initramfs modules vs userspace modules

Initramfs is not an opposite of userspace. Initramfs is a filesystem that contains tools and scripts run in userspace as well and mount real root filesystem.
Opposite of userspace is kernel space

meowxiik wrote:

I loaded the module during initramfs

Normally initramfs doesn't mount /boot because kernel and initramfs itself are already loaded into RAM by UEFI or bootloader. You are lucky that fat module was even present in initramfs.

meowxiik · 2024-12-30 03:19:46

dimich wrote:

Initramfs is not an opposite of userspace. Initramfs is a filesystem that contains tools and scripts run in userspace as well and mount real root filesystem.
Opposite of userspace is kernel space

I understand, but what do you call the stage after initramfs then? Arch Wiki describes it as "early" or "late userspace" which I figured was close enough.

https://wiki.archlinux.org/title/Arch_boot_process

meowxiik · 2024-12-30 03:21:15

dimich wrote:

You are lucky that fat module was even present in initramfs.

Oof, good to know

dimich · 2024-12-30 04:51:16

meowxiik wrote:

what do you call the stage after initramfs then? Arch Wiki describes it as "early" or "late userspace" which I figured was close enough.

I would say so, "late userspace" or "real rootfs stage". And "early userspace" or "initramfs stage" for opposite.

Arch Linux

#1 2024-12-29 13:19:53

[SOLVED] initramfs modules vs userspace modules

#2 2024-12-29 13:26:28

Re: [SOLVED] initramfs modules vs userspace modules

#3 2024-12-29 15:47:40

Re: [SOLVED] initramfs modules vs userspace modules

#4 2024-12-29 15:53:21

Re: [SOLVED] initramfs modules vs userspace modules

#5 2024-12-29 15:57:19

Re: [SOLVED] initramfs modules vs userspace modules

#6 2024-12-29 16:54:34

Re: [SOLVED] initramfs modules vs userspace modules

#7 2024-12-30 00:28:47

Re: [SOLVED] initramfs modules vs userspace modules

#8 2024-12-30 03:19:46

Re: [SOLVED] initramfs modules vs userspace modules

#9 2024-12-30 03:21:15

Re: [SOLVED] initramfs modules vs userspace modules

#10 2024-12-30 04:51:16

Re: [SOLVED] initramfs modules vs userspace modules

Board footer