Upgrade nightmare tonight...

zelurker · 2025-04-26 21:00:42

Switched to kernel 6.14.3, everything seemed to be fine, reboot, and there the new blue screen of kernel panic, which now requires you to have a cell phone to scan their stupid bar code, but anyway... ! Apparently it complains it can't execute init.
Well I am not an expert in initramfs images, but I haven't touched this stuff for months and everything worked fine until now, no idea what went wrong this time.
I rebooted with an usb key, but the generation of the intiramfs-linux.img seems ok, I was surprised when examining its contents using cpui -idv < initramfs-linux.img not to find any init inside, but looking at my laptop install it seems normal.

So after 1 hour spent on it, I am out of ideas. I need help on this!
The kernel panic screen you get by scanning the code doesn't give much info, no /init, so it tried some other possible locations, didn't find anything and so kernel panic.

I even tried to use a downgrade script to return to the previous kernel, it doesn't work anymore, same problem.

No mention of mounting anything in the boot log from the scanned logo, not the initramfs nor the rootfs, seeing it identified correctly the partitions, the initramfs must be right at least. ext4 is not in the fs modules anymore in recent kernels, but anyway I don't see anything about the root filesystem there.
I tried to pass root=/dev/sdX insted of the uuid, no change at all, still the blue kernel panic... !

And here is finally a link to the boot log, very very long link: https://panic.archlinux.org/panic_repor … 0603427550

Last edited by zelurker (2025-04-26 22:14:27)

zelurker · 2025-04-27 04:20:15

Ok, I was finally able to rescue it, but it was not easy : for an unknown reason, mkinitcpio stubbornly refuses to include any binary in the initramfs image on my desktop computer, no matter the config file.
But the laptop with the same version of mkinitcpio and the same config file for what I see can do it ! So I generated an initramfs-linux.img on the laptop, copied it on some usb stick, got it on the desktop as initramfs-linux2.img. It was not even for the same kernel, it was an older kernel on the laptop, but apparently it didn't need any module from the initramfs, but the binaries installed helped it tremendously to boot !

Well I'll keep this initramfs-linux2.img file around because for now it's the only way I have to boot my desktop pc in linux then !

I triied to install linux-lts too to see if it helped, but it was an initramfs generation problem so it produced a nice kernel panic too, without the bar code to scan though!

Last edited by zelurker (2025-04-27 04:23:05)

seth · 2025-04-27 05:36:14

mkinitcpio stubbornly refuses to include any binary in the initramfs image on my desktop computer, no matter the config file

Please post the mkinitcpio.conf and the output of mknitcpio

zelurker · 2025-04-27 08:39:33

Oh well quite standard, I tried udev and systemd, no difference, no binaries at all in both cases.
Currently:
MODULES=(ext4) # useless, there is no ext4 module in the recent kernel packages
BINARIES=()
FILES=()
HOOKS=(base systemd autodetect modconf keyboard keymap consolefont block filesystems fsck)
COMPRESSION="lzop"

and it displays no error at all, well except for consolefont and the missing firmwares):
manu@arch ~ $ sudo mkinitcpio -p linux
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img
==> Starting build: '6.14.3-arch1-1'
-> Running build hook: [base]
-> Running build hook: [systemd]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating lzop-compressed initcpio image: '/boot/initramfs-linux.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.14.3-arch1-1'
-> Running build hook: [base]
-> Running build hook: [systemd]
-> Running build hook: [modconf]
-> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: 'xhci_pci_renesas'
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'wd719x'
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating lzop-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful

Since both computers use the same version of mkinitcpio and the same configuration and only one of them puts binaries in the image I guess it's because of some external package, but no idea which one.

WorMzy · 2025-04-27 10:36:42

Please use code tags when posting terminal output or config file snippets.

https://wiki.archlinux.org/title/Genera … s_and_code

Mod note: not an installation issue. Moving to NC.

zelurker · 2025-04-27 10:39:12

Yeah well, it's an upgrade issue, so installation was close!
Sorry for the lack of tags, I have not used bbcode for years and won't use it for long anyway here again...

seth · 2025-04-27 11:15:18

and it displays no error at all, well except for consolefont and the missing firmwares):

So the initramfs gets generated, what does it look like afterwards?

lsinitcpio /boot/initramfs-linux.img

Assuming it's all fine the problem is most likely not "what", but "where"

lsblk -f
mount
cat /proc/cmdline

Last edited by seth (2025-04-27 11:15:36)

zelurker · 2025-04-27 11:27:09

seth wrote:

and it displays no error at all, well except for consolefont and the missing firmwares):
So the initramfs gets generated, what does it look like afterwards?
lsinitcpio /boot/initramfs-linux.img

That's interesting! I didn't know this lsinitcpio command, not an expert with cpio stuff... So the weird thing is that your lsinitcpio command displays some populated usr/bin directory but cpio -idv doesn't extract this directory, and the kernel neither, I just rebooted to be sure, and I got my kernel panic if using this initramfs file.

Assuming it's all fine the problem is most likely not "what", but "where"
lsblk -f
mount
cat /proc/cmdline

Well this info seems irrelevant but I can post if if you really want...

seth · 2025-04-27 11:35:43

Well this info seems irrelevant but I can post if if you really want...

The idea would be that you're storing the initramfs on the wrong partition.
If that's not the case for sure (ie. you generate the initramfs and overwrite the present one and things fail)

cat /boot/initramfs-linux.img | curl -F 'file=@-' 0x0.st # for the bogus initramfs

Let's have a look at it.
Also make sure you're not running out of space on /boot

zelurker · 2025-04-27 11:41:06

seth wrote:

Well this info seems irrelevant but I can post if if you really want...
The idea would be that you're storing the initramfs on the wrong partition.

Well /boot is on the root partition with plenty of free space, so I don't see what could go wrong here.

If that's not the case for sure (ie. you generate the initramfs and overwrite the present one and things fail)
cat /boot/initramfs-linux.img | curl -F 'file=@-' 0x0.st # for the bogus initramfs
Let's have a look at it.
Also make sure you're not running out of space on /boot

Here is the result: http://0x0.st/8WK3.cpio

seth · 2025-04-27 11:44:43

Hah, immediately ran into the overlooked

COMPRESSION="lzop"

Try the default zstd

zelurker · 2025-04-27 11:51:23

seth wrote:

Hah, immediately ran into the overlooked
COMPRESSION="lzop"
Try the default zstd

Ah sorry to disappoint you but I would have found if it was that easy ! ;-) I lost quite a few hours on this in the end...
Nope same result, lsinitcpio lists an /usr/bin directory which is not extracted by cpio -idv, I didn't try to reboot this time. I had already tried a few different compressions and no compression at all, I think the default in my configuration file was no compression at all actually.

seth · 2025-04-27 12:25:30

bin is a symlink to usr/bin but usr/bin isn't there, same goes for usr/lib… (as result the initcpio is also way too small, so it's not an extraction problem)

stat /usr/bin /usr/lib /usr/lib64

on the troublesome system.

zelurker · 2025-04-27 12:39:58

seth wrote:

bin is a symlink to usr/bin but usr/bin isn't there, same goes for usr/lib… (as result the initcpio is also way too small, so it's not an extraction problem)
stat /usr/bin /usr/lib /usr/lib64
on the troublesome system.

Yes sir !

  Fichier : /usr/bin
   Taille : 147456    	Blocs : 296        Blocs d'E/S : 4096   répertoire
Périphérique : 8/18	Inœud : 654084      Liens : 8
Accès : (0755/drwxr-xr-x)  UID : (    0/    root)   GID : (    0/    root)
 Accès : 2025-04-27 06:03:20.162456944 +0200
Modif. : 2025-04-27 06:03:09.204948142 +0200
Changt : 2025-04-27 06:03:09.204948142 +0200
  Créé : 2021-08-28 21:55:57.947856054 +0200
  Fichier : /usr/lib
   Taille : 294912    	Blocs : 584        Blocs d'E/S : 4096   répertoire
Périphérique : 8/18	Inœud : 654088      Liens : 292
Accès : (0755/drwxr-xr-x)  UID : (    0/    root)   GID : (    0/    root)
 Accès : 2025-04-27 05:50:43.182769030 +0200
Modif. : 2025-04-27 05:50:43.179435423 +0200
Changt : 2025-04-27 05:50:43.179435423 +0200
  Créé : 2021-08-28 21:55:57.947856054 +0200
  Fichier : /usr/lib64
   Taille : 4096      	Blocs : 8          Blocs d'E/S : 4096   répertoire
Périphérique : 8/18	Inœud : 1711476     Liens : 3
Accès : (0755/drwxr-xr-x)  UID : (    0/    root)   GID : (    0/    root)
 Accès : 2025-04-26 17:00:07.964466550 +0200
Modif. : 2024-08-09 04:50:48.853697392 +0200
Changt : 2025-04-15 16:47:45.846326986 +0200
  Créé : 2025-04-15 16:47:26.368568874 +0200

Scimmia · 2025-04-27 12:47:46

Why is /usr/lib64 not a symlink? Looks like you did something on the 15th that screwed a lot of things up.

Reinstall the filesystem package.

seth · 2025-04-27 12:49:43

And itr, also check the content of /usr/bin and see whether /bin is a symlink to /usr/bin

zelurker · 2025-04-27 12:54:34

seth wrote:

And itr, also check the content of /usr/bin and see whether /bin is a symlink to /usr/bin

itr ? What do you want to check in /usr/bin ? It's a big mess in there... and yes, /bin is a symlink to /usr/bin

zelurker · 2025-04-27 13:02:49

Scimmia wrote:

Why is /usr/lib64 not a symlink? Looks like you did something on the 15th that screwed a lot of things up.
Reinstall the filesystem package.

Very nice find ! It's actually while trying to install a cross compiler to test something, I thought the thing had only a separate directory in /usr, I didn't see it overwrote /usr/lib64 !
Oh well, just removed the whole tree because I had removed the rest of this cross compiler already, and reinstalled filesystem to be sure. (and made sure this /usr/lib64 just contained the cross compiler stuff).
Actually it was not done a long time ago.
Now I tried to generate a new initrd after that and it still has the same problem apparently... Hum, maybe I should try to reinstall a few packages after that then ?

seth · 2025-04-27 13:48:04

itr like "what happned to lib64 might have happened to /bin - but apparently not"

sudo mkinitcpio -v -p linux

zelurker · 2025-04-27 14:07:46

It's a huge log as you can imagine, so I posted this on pastebin : https://pastebin.com/kFSe3YMS
I had already a look at that of course, but it reports no error there, at least unless I missed something in all these lines... !

ps : I stopped when it wrote about generating the fallback image, the 1st one should be enough, right ?

Last edited by zelurker (2025-04-27 14:08:54)

seth · 2025-04-27 14:28:44

Right.
Interestingly, cpio doesn't extract /usr/bin here either (ie from my working intcpio)
"lsinitcpio -x" however suggests everything to be in place, got to check why my initramfs are obese.

Let's cycle back to the "where" question in #7 then

Arch Linux

#1 2025-04-26 21:00:42

Upgrade nightmare tonight...

#2 2025-04-27 04:20:15

Re: Upgrade nightmare tonight...

#3 2025-04-27 05:36:14

Re: Upgrade nightmare tonight...

#4 2025-04-27 08:39:33

Re: Upgrade nightmare tonight...

#5 2025-04-27 10:36:42

Re: Upgrade nightmare tonight...

#6 2025-04-27 10:39:12

Re: Upgrade nightmare tonight...

#7 2025-04-27 11:15:18

Re: Upgrade nightmare tonight...

#8 2025-04-27 11:27:09

Re: Upgrade nightmare tonight...

#9 2025-04-27 11:35:43

Re: Upgrade nightmare tonight...

#10 2025-04-27 11:41:06

Re: Upgrade nightmare tonight...

#11 2025-04-27 11:44:43

Re: Upgrade nightmare tonight...

#12 2025-04-27 11:51:23

Re: Upgrade nightmare tonight...

#13 2025-04-27 12:25:30

Re: Upgrade nightmare tonight...

#14 2025-04-27 12:39:58

Re: Upgrade nightmare tonight...

#15 2025-04-27 12:47:46

Re: Upgrade nightmare tonight...

#16 2025-04-27 12:49:43

Re: Upgrade nightmare tonight...

#17 2025-04-27 12:54:34

Re: Upgrade nightmare tonight...

#18 2025-04-27 13:02:49

Re: Upgrade nightmare tonight...

#19 2025-04-27 13:48:04

Re: Upgrade nightmare tonight...

#20 2025-04-27 14:07:46

Re: Upgrade nightmare tonight...

#21 2025-04-27 14:28:44

Re: Upgrade nightmare tonight...

Board footer