[SOLVED] Systemd-boot does not like initramfs-linux.img

samebutdifferent · 2015-07-17 16:10:09

Hello everyone,
today I decided to switch back from the LTS kernel to the official kernel. I installed the linux package and created a new entry for the boot loader:

root@laptop / # cat /boot/loader/entries/arch.conf 
title   Arch Linux
linux   vmlinuz-linux
initrd  intel-ucode.img
initrd  initramfs-linux.img
options root=PARTUUID=97dbf76a-a32e-454e-a4f3-1c66a887d45c rootflags=defaults,relatime,discard rw quiet

Unfortunately after a reboot all I got was:

Failed to open file: initramfs-linux.img
Trying to load files to higher address
Failed to open file: initramfs-linux.img

Since this is the same entry that I use for the LTS kernel (excluded the filenames) it must works.
Out of ideas, I copied "initramfs-linux.img" to "initramfs-linux.img2", rebooted and, at the boot menu, edited the command line changing initrd to point to "initramfs-linux.img2". Surprisingly the system booted correctly.

I have no idea why I can't use "initramfs-linux.img". What can be happening ?.

As you can see "initramfs-linux.img2" is the same:

root@laptop boot # stat initramfs-linux.img
  File: ‘initramfs-linux.img’
  Size: 3621065   	Blocks: 7073       IO Block: 512    regular file
Device: 801h/2049d	Inode: 19          Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-07-17 02:00:00.000000000 +0200
Modify: 2015-07-17 16:30:36.000000000 +0200
Change: 2015-07-17 16:30:36.000000000 +0200
 Birth: -
root@laptop boot # stat initramfs-linux.img2
  File: ‘initramfs-linux.img2’
  Size: 3621065   	Blocks: 7073       IO Block: 512    regular file
Device: 801h/2049d	Inode: 22          Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-07-17 02:00:00.000000000 +0200
Modify: 2015-07-17 16:37:10.000000000 +0200
Change: 2015-07-17 16:37:10.000000000 +0200
 Birth: -
root@laptop boot # ls -al
total 54792
drwxr-xr-x  4 root root     1024 Jan  1  1970 .
drwxr-xr-x 17 root root     4096 Jun 16 10:15 ..
drwxr-xr-x  3 root root      512 Jul 16 14:21 EFI
-rwxr-xr-x  1 root root 18528996 Jul 17 16:30 initramfs-linux-fallback.img
-rwxr-xr-x  1 root root  3621065 Jul 17 16:30 initramfs-linux.img
-rwxr-xr-x  1 root root  3621065 Jul 17 16:37 initramfs-linux.img2
-rwxr-xr-x  1 root root 17959271 Jul 12 18:55 initramfs-linux-lts-fallback.img
-rwxr-xr-x  1 root root  3582355 Jul 12 18:54 initramfs-linux-lts.img
-rwxr-xr-x  1 root root   663040 Feb  9 16:42 intel-ucode.img
drwxr-xr-x  3 root root      512 Jul 16 14:04 loader
-rwxr-xr-x  1 root root  4221920 Jul 15 10:32 vmlinuz-linux
-rwxr-xr-x  1 root root  3901536 Jul 10 21:14 vmlinuz-linux-lts
root@laptop boot # df -m .
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sda1            127    54        73  43% /boot

Notes:
I'm using EFI so the hard disk is partitioned in GPT and the boot partition is vfat; fsck do not reports any errors.
I already did a clean reinstall of mkinitcpio and recreated the images many times.
I'm using an 840EVO SSD with the old firmware (not the new one that caused corruption on linux a while ago).

Last edited by samebutdifferent (2015-07-19 18:45:17)

Scimmia · 2015-07-17 16:13:22

You are simply giving it filenames when it needs paths (relative to the ESP).

samebutdifferent · 2015-07-17 16:24:24

You mean that I should add backslashes ?

root@laptop entries # cat arch.conf 
title   Arch Linux
linux   \vmlinuz-linux
initrd  \intel-ucode.img
initrd  \initramfs-linux.img
options root=PARTUUID=97dbf76a-a32e-454e-a4f3-1c66a887d45c rootflags=defaults,relatime,discard rw quiet

That does not change the outcome...

Scimmia · 2015-07-17 16:33:19

*nix doesn't use backslashes for paths.

If simply renaming it works, that's probably not the issue. I guess it assumes the root of the ESP if no path is given?

Try regenerating the initramfs.

Last edited by Scimmia (2015-07-17 16:39:55)

samebutdifferent · 2015-07-17 16:58:57

Scimmia wrote:

*nix doesn't use backslashes for paths.

Yes, but according to the documentation EFI need those.

Scimmia wrote:

Try regenerating the initramfs.

I have already did that a couple times ("mkinitcpio -p linux" right ?).

Scimmia · 2015-07-17 17:06:54

The nvram entry may need backslashes, but systemd-boot would use forward slashes.

I keep asking things you've already tried, don't I? I really need to slow down and read better.

I wonder if deleting it then regenerating it would make a difference? It would change some things in the filesystem that wouldn't be changed by just overwriting it.

samebutdifferent · 2015-07-17 17:44:16

Deleted and regenerated, but "initramfs-linux.img" is still not visible by the boot loader.
In addition now I'm using the correct slashes.

Head_on_a_Stick · 2015-07-17 22:03:23

Can you boot via a direct NVRAM entry?

# efibootmgr -d /dev/sda -p 1 -c -L "Arch" -l '/vmlinuz-linux' -u "root=PARTUUID=97dbf76a-a32e-454e-a4f3-1c66a887d45c rw quiet initrd=/intel-ucode.img initrd=/initramfs-linux.img"

Select this entry from your UEFI boot menu.

By the way, NVRAM entries will accept either forward- or back-slashes

samebutdifferent · 2015-07-17 23:03:24

Head_on_a_Stick wrote:

Can you boot via a direct NVRAM entry?
# efibootmgr -d /dev/sda -p 1 -c -L "Arch" -l '/vmlinuz-linux' -u "root=PARTUUID=97dbf76a-a32e-454e-a4f3-1c66a887d45c rw quiet initrd=/intel-ucode.img initrd=/initramfs-linux.img"
Select this entry from your UEFI boot menu.

Unfortunately I can't boot directly using efistub. If I could I wouldn't be using systemd-boot.
I think the reason is that the arguments (-u ...) are not passed to the loader (-l ...) because of a bug in the UEFI firmware of my laptop (dell e5420, from 2011).
What I see when I run your entry is: kernel panic unable to mount root fs on unknown-block(0 0) ...

samebutdifferent · 2015-07-18 12:30:17

Today I explored all options in my bios and found that it shows my initramfs with an old dos name for some unknow reason.
Since removing and renaming has no effect, I formatted the partiton and regenerated everything.
Now "initramfs-linux.img" can be loaded correctly but not "initramfs-linux-fallback.img" because it's viewed as "INITRA~2.IMG" ...

Note that the initramfs for the LTS version are all correct ... What can be happening to that particular filename ?

Last edited by samebutdifferent (2015-07-18 12:31:07)

Lone_Wolf · 2015-07-18 14:37:26

This probably has do to with the FAT filesystem.

Every file with a long filename (LFN) has a corresponding 8.3 entry in the FAT-table .
VFAT takes care of keeping the connection between the 8.3 entry and it's LFN counterpart intact.

Sometimes creating/copying breaks that connection and the link between the 8.3 FAT-entry and it's LFN-entry is gone.
Then the only working reference to the file is the 8.3 entry.

The VFAT problems could be caused by mobo firmware, SSD firmware or the linux kernel.

samebutdifferent · 2015-07-18 17:41:10

Lone_Wolf wrote:

Sometimes creating/copying breaks that connection and the link between the 8.3 FAT-entry and it's LFN-entry is gone.
Then the only working reference to the file is the 8.3 entry.

I don't think the problem is a broken reference because I can rename or remove both the long and the short filename without problems.

Lone_Wolf · 2015-07-19 15:18:46

samebutdifferent, I'll try to be more clear .

FAT system uses 1 table that links 8.3 filenames with their physical location.
ALL files on a FAT system can only be accessed through the underlying FAT table that ONLY holds 8.3 filenames.

VFAT maintains a separate table that links LFN entries with their 8.3 counterpart.
basically any LFN will be shortened to fit in an 8.3 scheme, VFAT will create that shortened entry in the FAT-Table if it doesn't exist yet.
VFAT handles both LFN and 8.3 filenames.

Vfat-aware code will always call the VFat routines to access any file, regardless of LFN or 8.3

However, any non-vfat aware code calls the fat routines directly, messing things up.
Buggy VFAT code can also have this effect.

The screenshot you posted clearly shows an INITRA~2.IMG , this is the 8.3 reference created for an LFN entry.
At some point VFAT/FAT connection for that specific LFN got messed up.

The problem is detected during boot, likely by systemd-boot code, but can have been caused by any code accessing the ESP.
Troubleshooting this further won't be easy.

Last edited by Lone_Wolf (2015-07-19 15:20:14)

samebutdifferent · 2015-07-19 18:12:32

I think I got it.
After a lot of tests I found that the error occurs when I create consecutively 5 long filenames that begin in the same way.
It seems the crappy firmware of my Dell can't even read correctly the FAT partition (in addition to not passing parameters to the EFI executables)

In the end I decided to remove linux-lts completely and keep only 2 initramfs to not trigger the problem.

If someone feels to try to recreate the problem on his computer, post below the results, I'm curious to know if this happens on others brands.

progandy · 2015-07-19 18:50:35

samebutdifferent wrote:

I think I got it.
After a lot of tests I found that the error occurs when I create consecutively 5 long filenames that begin in the same way.
It seems the crappy firmware of my Dell can't even read correctly the FAT partition (in addition to not passing parameters to the EFI executables)
In the end I decided to remove linux-lts completely and keep only 2 initramfs to not trigger the problem.

You could also change the naming scheme for these files. mkinitcpio reads the names from /etc/mkinitcpio.d/*.preset

mirh · 2016-04-09 19:39:23

Lone_Wolf wrote:

Troubleshooting this further won't be easy.

Indeed it hasn't been here.
I cross-checked plenty of times ESP partition: mtools, this script, in windows.
They all retrieved consistently both short and long names.

And I could rule out it was something with systemd-boot, since it also happened with the straightforward EFISTUB.

Then I started to wonder why GRUB worked on the other hand.
And albeit I was sure it was using the same options of systemd-boot, I spotted from live system that the kernel was receiving completely different parameters.
I realized then those must have been specific options, probably related to its bios roots, which in turn would explain why it doesn't even pass (and need) initrd flag.
And last, in its rescue shell ls still managed to list files correctly. And here's when I noticed that GRUB had loaded its own modules to handle fat32.

Contemporarily, I dd-ed whole /boot partition and I tried to check if VMWare Workstation own efi shell experienced the problem
The answer was negative. Then the epiphany:

... TL;DR: I started EFI shell on my laptop (shellx64.efi), loaded EDK2 driver, unloaded AMI File System Driver, connected FAT File System Driver and.. Worked!
I could even exit shell and boot with the normally malfunctioning entries.

...
Now, I sit here, after I extracted said allegedly broken driver wondering what to do.

EDIT: ideas
EDIT2: and another
EDIT3: hacking around uefi modules seems really easy it must be said...

EDIT4: anyway, AMI seems to have fixed the bug somewhere in 2012 (or at least I noticed such effect when switching from asus K53U 223 bios to K53BE 212)

Last edited by mirh (2018-10-03 17:33:39)

Arch Linux

#1 2015-07-17 16:10:09

[SOLVED] Systemd-boot does not like initramfs-linux.img

#2 2015-07-17 16:13:22

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#3 2015-07-17 16:24:24

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#4 2015-07-17 16:33:19

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#5 2015-07-17 16:58:57

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#6 2015-07-17 17:06:54

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#7 2015-07-17 17:44:16

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#8 2015-07-17 22:03:23

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#9 2015-07-17 23:03:24

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#10 2015-07-18 12:30:17

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#11 2015-07-18 14:37:26

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#12 2015-07-18 17:41:10

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#13 2015-07-19 15:18:46

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#14 2015-07-19 18:12:32

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#15 2015-07-19 18:50:35

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

#16 2016-04-09 19:39:23

Re: [SOLVED] Systemd-boot does not like initramfs-linux.img

Board footer