[SOLVED] efibootmgr: ESP not recognized

fatal_error · 2020-01-06 17:43:58

Dear all,

I've followed the Arch installation guide to install a new System - including FDE, having /boot on a separate USB stick and booting via UEFI directly. The FDE part works, but I seem to run into trouble regarding my EFI System Partition (ESP), which is the /boot partition as well in my case. I can successfully manipulate EFI boot entries via efibootmgr (add entries, change boot order etc.). However, upon restart the EFI partition is not recognized by my System. This is odd, as the UEFI software on my board is usually smart enough to detect all ESPs on all attached USB sticks (as it does for the Arch installation USB stick) and present them in the Boot Menu. I'm a bit stumped on how to proceed, the following "oddities" come to mind with my ESP, not sure if any of them pose a problem:

1. The ESP is rather large: it takes up the entire 40 GB of the Stick (formatted as FAT32). Both the "boot" and "esp" flags are set for the partition.
2. the EFI/arch directory was created manually.
3. vmlinuz-linux.efi is really just the Kernel copied from /boot and renamed with a ".efi" extension - is that correct? AFAIK that should be enough for EFI Stub kernels, right?

For the sake of completeness, these are the contents of my /boot, all of which reside on an external USB stick - comments are in parantheses:

ll -R
.:
drwxr-xr-x  3 root root    16K Jan  1  1970 ./
drwxr-xr-x 24 root root     4K Dec  3 07:56 ../
drwxr-xr-x  3 root root    16K Jan  6 16:13 EFI/
-rwxr-xr-x  1 root root  8192K Jan  5 21:15 luks_header.img* (the detached LUKS header of the encrypted partition on the main drive)
-rwxr-xr-x  1 root root 65921K Jan  6 13:05 initramfs-linux-fallback.img*
-rwxr-xr-x  1 root root 46839K Jan  6 13:05 initramfs-linux.img*
-rwxr-xr-x  1 root root  3048K Dec 11 13:32 intel-ucode.img* (Intel Microcode)
-rwxr-xr-x  1 root root  1024K Jan  5 21:06 luks_key.img* (Key file to unlock the LUKS partition) 
-rwxr-xr-x  1 root root  6243K Jan  6 11:09 vmlinuz-linux*

./EFI:
drwxr-xr-x 3 root root 16K Jan  6 16:13 ./
drwxr-xr-x 3 root root 16K Jan  1  1970 ../
drwxr-xr-x 2 root root 16K Jan  6 17:32 arch/

./EFI/arch: (This directory was created manually and contains copy of the boot root. 
drwxr-xr-x 2 root root    16K Jan  6 17:32 ./
drwxr-xr-x 3 root root    16K Jan  6 16:13 ../
-rwxr-xr-x 1 root root 65921K Jan  6 13:05 initramfs-linux-fallback.img*
-rwxr-xr-x 1 root root 46839K Jan  6 13:05 initramfs-linux.img*
-rwxr-xr-x 1 root root  3048K Dec 11 13:32 intel-ucode.img*
-rwxr-xr-x 1 root root  6243K Jan  6 11:09 vmlinuz-linux.efi* (Copied kernel with .efi extension)

Am I missing something obvious here? What I really don't get is why the device does not show up as a boot device at all - this works e.g. for the Arch Installation USB stick, so why would it not work here? Thanks for your help.

Last edited by fatal_error (2020-01-08 10:19:56)

Head_on_a_Stick · 2020-01-06 17:48:01

Please post the output of

efibootmgr -v
parted --list

And also the full efibootmgr command that you used.

If /boot is already on the USB stick why have you copied the kernel image? EFI_STUB booting does not require that the kernel image be named with an .efi suffix.

fatal_error · 2020-01-06 19:42:33

I thought the "EFI" directory was required by the EFI specification, but if I'm mistaken there I'll gladly remove it again. However, the ESP is not recognized with or without the EFI directory and its contents. This is the command that was used to create the boot entry:

efibootmgr --disk /dev/sda --part 1 --create --label "Linux" --loader /vmlinuz-linux --unicode 'root=/dev/cryptvolgroup/root rw initrd=\EFI\arch\intel-ucode.img initrd=\EFI\arch\initramfs-linux.img' --verbose

parted --list

Model: USB Flash Disk (scsi)
Disk /dev/sda: 40.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name        Flags
 1      1049kB  40.0GB  40.0GB  fat32        EFI System  boot, esp


Model: SanDisk Ultra (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size    File system  Name                          Flags
 1      1049kB  556MB  555MB   ntfs          Basic data partition          hidden, diag
 2      556MB   661MB  105MB   fat32        EFI system partition          boot, esp
 3      661MB   677MB 16.8MB                   Microsoft reserved partition  msftres
 4      677MB    51GB     50GB   ntfs           Basic data partition          msftdata
 5      51GB    512GB   461GB                    Linux

efibootmgr -v

BootCurrent: 0005
Timeout: 2 seconds
BootOrder: 0005,0000
Boot0000* Windows Boot Manager	HD(2,GPT,1f13fb08-30bc-11ea-978f-2e728ce88125,0x101000,0x64000)/File(\EFI\MICROSOFT\BOOT\BOOTMGFW.EFI)WINDOWS.........x...
Boot0005* Arch Linux	HD(1,GPT,52df3092-30bc-11ea-aec2-2e728ce88125,0x800,0x2b3f7e7)/File(\vmlinuz-linux)r.o.o.t.=./.d.e.v./.c.r.y.p.t.v.o.l.g.r.o.u.p./.r.o.o.t. .r.w. .i.n.i.t.r.d.=.\.E.F.I.\.a.r.c.h.\.i.n.t.e.l.-.u.c.o.d.e...i.m.g. .i.n.i.t.r.d.=.\.E.F.I.\.a.r.c.h.\.i.n.i.t.r.a.m.f.s.-.l.i.n.u.x...i.m.g.

As said, the EFI boot menu only recognizes Windows as bootable, no other boot options/devices are selectable.

Head_on_a_Stick · 2020-01-06 20:00:50

Does it boot if you share the ESP on /dev/sdb instead of using a USB stick?

I can't find anything about the maximum size of an ESP but the implementations are so buggy in general that it wouldn't surprise me if the 40GiB size was b0rking things.

Lone_Wolf · 2020-01-07 11:36:48

https://en.wikipedia.org/wiki/File_Allo … imum_sizes

40 GiB is possible, but windows tools won't allow you to create fat32 volumes above 32 GiB .

No idea if the ESP specification lists a maximum size, but I agree with head_on_a_stick that I'd be surprised to find any efi firmware that supports > 32 GiB .

Last edited by Lone_Wolf (2020-01-07 11:37:57)

fatal_error · 2020-01-07 11:46:27

Head_on_a_Stick and Lone_Wolf, thank you both for your answers/the provided link. I too agree that the EFI partition size might be the culprit here. That's why I also mentioned the partition size in my original post. It was just that I was thinking "heck it's 2020, that cannot be a problem anymore" - but who knows ;-) I'll try your suggestions tonight and come back to you with the results.

fatal_error · 2020-01-07 13:50:58

A reply to Head_on_a_Stick:

Head_on_a_Stick wrote:

If /boot is already on the USB stick why have you copied the kernel image? EFI_STUB booting does not require that the kernel image be named with an .efi suffix.

I got the impression that an esp/EFI/ path was required from this article: https://wiki.archlinux.org/index.php/EFISTUB

Linux Kernel EFISTUB initramfs path should be relative to the EFI System Partition's root and use backslashes (in accordance with EFI standards). For example, if the initramfs is located in esp/EFI/arch/initramfs-linux.img, the corresponding UEFI formatted line should be initrd=\EFI\arch\initramfs-linux.img. In the following examples we will assume that everything is under esp/

Is this correct? "esp" in my case would be /boot, hence the path "/boot/EFI/arch/[kernel and initramdisk goes here].

Head_on_a_Stick · 2020-01-07 14:51:14

fatal_error wrote:

I got the impression that an esp/EFI/ path was required from this article: https://wiki.archlinux.org/index.php/EFISTUB
[...]
Is this correct?

The article is correct (except in respect of the backslashes: forward slashes work fine too) but the paths were given as examples rather than an absolute instruction. All that is required is that the kernel image & initramfs be on the EFI system partition and that the correct path is called in the efibootmgr command, you can put them wherever you want. If you choose to copy them somewhere else then you'll have to write a pacman hook to re-copy them when the kernel is updated, which seems like a lot of effort for no return.

fatal_error · 2020-01-07 16:06:55

Head_on_a_Stick wrote:

fatal_error wrote:
I got the impression that an esp/EFI/ path was required from this article: https://wiki.archlinux.org/index.php/EFISTUB
[...]
Is this correct?
The article is correct (except in respect of the backslashes: forward slashes work fine too) but the paths were given as examples rather than an absolute instruction. All that is required is that the kernel image & initramfs be on the EFI system partition and that the correct path is called in the efibootmgr command, you can put them wherever you want. If you choose to copy them somewhere else then you'll have to write a pacman hook to re-copy them when the kernel is updated, which seems like a lot of effort for no return.

OK,

thanks for the clarification on this - yes, copying the kernel and RAM disk seems like unnecessary work that I would like to avoid - totally agree.
I'm still at the office and tried with my now modified boot stick, but alas the stick is not recognized as a valid boot option here as well (will try later at home on the target machine). The stick now looks like as follows:

* /dev/sda1 is now 512 MB in size, FAT32-formatted and contains the ESP. The flags "esp" and "boot" are set in parted. The partition type says "EFI System Partition". The contents of the partition are the same as listed in my original post.
* The rest of the space is assigned to a (now empty) ext2 partition.

As mentioned, also with this setup the Stick is not recognized (at least here at work with UEFI Boot enabled). Just to make sure, the very same Stick was used to install Ubuntu, so it should not be a hardware issue. What I did not do here at work was adding a new boot entry via efibootmgr - so obviously the Kernel would not boot on this machine, I'm expecting some Kernel panic or early exception due to the Initial RAM disk not being extracted. However, I'm not even getting to that point - the USB Stick is not presented in the UEFI Boot menu, although booting from USB is enabled in the UEFI. From my understanding, for a one-time boot it should not be necessary to issue the efibootmgr command at all. UEFI should be "smart enough" to find any bootable executables on attached USB mass storage devices - right? I mean thats how the Install USB sticks are being recognized as bootable - correct? I'm getting the feeling I am missing something very basic here but cannot figure out what it is ;-)

Last edited by fatal_error (2020-01-07 16:08:10)

Head_on_a_Stick · 2020-01-07 16:26:48

fatal_error wrote:

the stick is not recognized as a valid boot option here

Secure boot?

fatal_error wrote:

From my understanding, for a one-time boot it should not be necessary to issue the efibootmgr command at all. UEFI should be "smart enough" to find any bootable executables on attached USB mass storage devices - right?

To boot a USB stick without any NVRAM entries copy the loader to the EFI system partition at /EFI/Boot/bootx64.efi: https://www.rodsbooks.com/efi-bootloade … ive-naming

But that won't work for the kernel unless you compile a custom version with CONFIG_CMDLINE set to declare the root partition & initramfs location.

fatal_error · 2020-01-08 10:19:09

Secure boot is disabled. I found the same information you provided regarding the fallback /EFI/BOOT/bootx64.efi path. Of course this does not work, for the reasons you pointed out. However, I finally got it to work yesterday evening. These are the steps I tried:

1. efibootmgr gladly manipulated boot entries, but all entries where gone upon reboot - so no changes were stored in NVRAM.
2. Next I tried to add boot entries via the UEFI v2 shell on the Arch Installer ISO. Same picture: adding boot entries worked like a charm, still they were gone after a reboot.
3. The final desperate attempt to boot via UEFI was adding a startup.nsh script at the root of the ESP as described here: https://wiki.archlinux.org/index.php/EFISTUB#More_tools. Still, the UEFI refused to even detect the USB stick as bootable.
4. At this point I had given up the "UEFI-only" route as my EFI firmware seems to be complete and utter crap - thanks for that MSI. So I decided to install a simple boot manager, which brought me to the rEFInd wiki entry: https://wiki.archlinux.org/index.php/REFInd. The installer script did not do anything for me, fortunately manual configuration is very easy and straight-forward. I put the rEFInd loader under the EFI fallback boot path (esp/EFI/BOOT/bootx64.efi). I decided not to create a separate refind-linux.conf file, I did all my manual configuration in the esp/EFI/refind/refind.conf file. Rebooted, unlocking the LUKS partition with the detached header and key file worked on the first attempt. Now I'm sitting at my root shell, smiling happily ;-)

Thanks again for all the support, I hope this post helps future users that attempt UEFI boots and fail due to buggy implementations as I did.

Arch Linux

#1 2020-01-06 17:43:58

[SOLVED] efibootmgr: ESP not recognized

#2 2020-01-06 17:48:01

Re: [SOLVED] efibootmgr: ESP not recognized

#3 2020-01-06 19:42:33

Re: [SOLVED] efibootmgr: ESP not recognized

#4 2020-01-06 20:00:50

Re: [SOLVED] efibootmgr: ESP not recognized

#5 2020-01-07 11:36:48

Re: [SOLVED] efibootmgr: ESP not recognized

#6 2020-01-07 11:46:27

Re: [SOLVED] efibootmgr: ESP not recognized

#7 2020-01-07 13:50:58

Re: [SOLVED] efibootmgr: ESP not recognized

#8 2020-01-07 14:51:14

Re: [SOLVED] efibootmgr: ESP not recognized

#9 2020-01-07 16:06:55

Re: [SOLVED] efibootmgr: ESP not recognized

#10 2020-01-07 16:26:48

Re: [SOLVED] efibootmgr: ESP not recognized

#11 2020-01-08 10:19:09

Re: [SOLVED] efibootmgr: ESP not recognized

Board footer