Cannot boot Arch from systemd-boot installation with XBOOTLDR

emk2203 · 2024-10-17 17:40:23

Pretext: I have done several Arch installations now with a systemd-boot installation, XBOOTLDR partition and discoverable partitions Discoverable Partitions .

There have been no issues until my new PC, a ASUS B660Plus Wifi D4. I can setup the machine as usual (it's a double-boot with Windows 11), and after installation, I get the systemd-boot menu. Except that the Arch entry is missing. I can boot to Windows and to the BIOS setup (automatically generated entries). I have tried several things recommended in the Wiki, like disabling Fast Boot, also disabling Intel RST/VMD to make sure that the disks (3x NVMe) are passed through without getting captured by a RAID controller.

Whatever I do, I can't get user entries for systemd-boot to show up, only the autogenerated ones. From a rescue system with `gdisk`:

Number  Start (sector)    End (sector)  Size       Code  Name
   1             256           25855   100.0 MiB   EF00  EFI system partition
   2           25856           29951   16.0 MiB    0C01  Microsoft reserved ...
   3           29952        25402111   96.8 GiB    0700  Basic data partition
   4        25402112        25599999   773.0 MiB   2700
   5        25600000        25676799   300.0 MiB   EA00  XBOOTLDR partition
   6        25676800        27571199   7.2 GiB     8200  Linux swap
   7        27571200        34124799   25.0 GiB    8304  Linux x86-64 root (/)
   8        34124800        47231999   50.0 GiB    8302  Linux /home
   9        47232000        53785599   25.0 GiB    8300  Linux filesystem
  10        53785600        60339199   25.0 GiB    8300  Linux filesystem
  11        60339200       488378623   1.6 TiB     8306  Linux /srv

codes for the partitions

nvme0n1                                                                                       
|-nvme0n1p1  vfat   FAT12                 4EB0-AF32                              68.7M    31% /mnt/install/efi
|-nvme0n1p2                                                                                   
|-nvme0n1p3  ntfs         Win11OS         641843BC18438BCC                                    
|-nvme0n1p4  ntfs                         DC76F47776F453AC                                    
|-nvme0n1p5  vfat   FAT32                 C734-993C                             227.8M    24% /mnt/install/boot
|-nvme0n1p6  swap   1     SWAP            21ad2ad6-5181-4b78-b813-19a4e953269b                [SWAP]
|-nvme0n1p7  ext4   1.0                   19f696d5-ba9a-4acc-8f48-f8cff2073d83   18.7G    18% /mnt/install
|-nvme0n1p8  ext4   1.0                   51ba5e58-2371-4ec8-847c-593ad9434783   46.4G     0% /mnt/install/home
|-nvme0n1p9                                                                                   
|-nvme0n1p10                                                                                  
`-nvme0n1p11 ext4   1.0                   ef54b389-59ad-45c2-89e7-33b3791898d4    1.5T     0% /mnt/install/srv

filesystems on the partitions

boot
|-- initramfs-linux.img
|-- intel-ucode.img
|-- loader
|   |-- entries
|   |   |-- archlinux-core-main.conf
|   |   `-- memtest.conf
|   |-- entries.srel
|   |-- loader.conf
|   `-- random-seed
|-- memtest86+
|   `-- memtest.efi
`-- vmlinuz-linux

structure of `/boot`

[root@archboot /mnt/install]# tree -L 3 efi
efi
|-- EFI
|   |-- Boot
|   |   `-- bootx64.efi
|   |-- Microsoft
|   |   |-- Boot
|   |   `-- Recovery
|   `-- systemd
|       `-- systemd-bootx64.efi
`-- shellx64.efi

7 directories, 3 files

structure of `/efi`

[root@archboot /mnt/install]# cat boot/loader/entries/archlinux-core-main.conf 
title    Arch Linux
linux    /vmlinuz-linux
initrd   /intel-ucode.img
initrd   /initramfs-linux.img
options  quiet loglevel=3 consoleblank=300

contents of `archlinux-core-main.conf` (to boot Arch)

[root@archboot /mnt/install]# cat boot/loader/loader.conf                      
default archlinux-core-main
timeout 5
console-mode keep
#default @saved

contents of `loader.conf` (systemd-boot loader configuration)

Please let me know what information is missing.

I also tried once to give up on the Discoverable Partitions and boot Arch with systemd-boot and the UUID in the options line (`options root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx rw`), but this didn't work either.

Any ideas?

cryptearth · 2024-10-17 17:52:21

although I'm no expert in systemd-boot as I use grub when tested in a vm I used ext4 for the xbootldr partition and had to add the driver from the efifs package
but as uefi is meant to support fat and there's no fat fs driver in efifs I guess an issue could be the xbootldr partition is skipped
can you find the xbootldr partition by calling map from with the efi shell? if it doesn't show up it's not loaded properly
also: how EXACTLY do you install bootctl?

emk2203 · 2024-10-17 18:37:51

After I mounted all partitions to their respective places, I arch-chroot into the system and then use

bootctl --esp-path=/efi --boot-path=/boot install

.

XBOOTLDR is supposed by the spec to have vfat, for the same reasons as the EFI partition (needs to be accessible). This is why it's vfat and therefore needs no additional drivers. EFI partition is also accessible w/o additional drivers.

I try the map command later and edit this then.

V1del · 2024-10-17 18:39:13

Agreed with the above. Why are you using FAT12 ? Chances are mainboard vendors (i.e. ASUS) don't explicitly support this and without a FAT driver in efifs, if the UEFI says it doesn't know this filesystem because they limit themselves to FAT32 you're not going to find the partition.

emk2203 · 2024-10-17 21:04:10

V1del wrote:

Agreed with the above. Why are you using FAT12 ? Chances are mainboard vendors (i.e. ASUS) don't explicitly support this and without a FAT driver in efifs, if the UEFI says it doesn't know this filesystem because they limit themselves to FAT32 you're not going to find the partition.

I think this came from the Windows setup. I definitely didn't set this on purpose. Anyway, the system boots to MSWindows. The UEFI should know about the format then, otherwise Windows wouldn't boot from this partition if the UEFI couldn't find it at all.

But I can back up the contents and reformat / restore; maybe worth a try.

cryptearth · 2024-10-18 06:37:56

@v1del
the current fat12 with the size of just 100mb is what the windows installer creates unless you do it manual with diskpart and bcdboot

however: a point often came up lately it seems many oem design and test thier firmware windows only and due to untested bugs breaks tge moment something different shows up - so I see potential issues:
- the uefi just skips all other partitions because it already found its ESP
- similar to the above: the uefi can't handle more than one fat partition correctly and expects the esp to be the first
in both cases using a different filesystem for xbootldr partition and a driver from efifs package might could do the trick
I know this is not how it's supposed to work - but with windows still holding more than 80% marketshare there's no incentive for any of the big oem to verify thier firmwares with linux
if you want that level of support you have to move from consumer hardware to those targeted to professionals and business like asrock rack or supermicro or similar

emk2203 · 2024-10-18 08:56:37

Since it was requested, here the mappings from the UEFI shell:

Mapping table
      FS0: Alias(s):HD0x0b0b:;BLK1:
          PciRoot(0x0)/Pci(0x14,0x0)/USB(0x17,0x0)/USB(0x1,0x0)/HD(1,GPT,52FE17B9-E327-1199-F10F-3459C020493B,0x800,0x5B3028F)
      FS1: Alias(s):HD0x0b0c:;BLK2:
          PciRoot(0x0)/Pci(0x14,0x0)/USB(0x17,0x0)/USB(0x1,0x0)/HD(2,GPT,C4678A72-7419-B328-F346-EB2B55769E66,0x5B30A8F,0x10000)
      FS4: Alias(s):HD3b:;BLK10:
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(1,GPT,48DC55E3-95F8-4463-B690-55030B5E2027,0x100,0x6400)
      FS5: Alias(s):HD3d:;BLK14:
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(3,GPT,2D2431A2-1029-4F83-9C1E-6D1CBE016B63,0x7500,0x1832600)
      FS6: Alias(s):HD3e:;BLK15:
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(4,GPT,7B3B536E-3E3E-4188-A007-A4470CD340D6,0x1839B00,0x30500)
      FS7: Alias(s):HD3f:;BLK16:
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(5,GPT,861F195D-EA20-4B6C-A73B-A84F53DB07C8,0x186A000,0x12C00)
      FS2: Alias(s):HD1c:;BLK6:
          PciRoot(0x0)/Pci(0x1A,0x0)/Pci(0x0,0x0)/NVMe(0x1,B8-77-91-31-53-38-25-00)/HD(2,GPT,8C191AC8-FC76-4394-BB18-F47B78547AFC,0x8000,0xE8E00000)
      FS3: Alias(s):HD2b:;BLK8:
          PciRoot(0x0)/Pci(0x1D,0x0)/Pci(0x0,0x0)/NVMe(0x1,45-D8-DB-2A-01-75-A0-00)/HD(1,GPT,D38CCCAE-E7D5-46A0-8584-CD7DE407722D,0x800,0xE8E08000)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x14,0x0)/USB(0x17,0x0)/USB(0x1,0x0)
     BLK3: Alias(s):
          PciRoot(0x0)/Pci(0x14,0x0)/USB(0x2,0x0)
     BLK9: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)
    BLK13: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(2,GPT,41D09BE1-397A-4493-A357-B73CF986DC94,0x6500,0x1000)
    BLK17: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(6,GPT,036FC676-8762-4FC3-AFF5-40EF60BBCBB3,0x187CC00,0x1CE800)
    BLK18: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(7,GPT,D4AB315B-2AC8-44BA-B5A0-515F0CBF8DA6,0x1A4B400,0x640000)
    BLK19: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(8,GPT,4C98BD69-BBC1-43DC-AE65-3D2FA1972CB0,0x208B400,0xC80000)
    BLK20: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(9,GPT,7CE19987-4698-4C6C-A9B9-AD7BB9948CEA,0x2D0B400,0x640000)
    BLK11: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(10,GPT,398B4473-B44F-421A-8FB6-2866A8B1E1DB,0x334B400,0x640000)
    BLK12: Alias(s):
          PciRoot(0x0)/Pci(0x6,0x0)/Pci(0x0,0x0)/NVMe(0x1,F9-91-95-45-8B-44-1B-00)/HD(11,GPT,0CCB6850-7DC4-47CB-90EC-DFC7E53E1B95,0x398B400,0x19835D00)
     BLK4: Alias(s):
          PciRoot(0x0)/Pci(0x1A,0x0)/Pci(0x0,0x0)/NVMe(0x1,B8-77-91-31-53-38-25-00)
     BLK5: Alias(s):
          PciRoot(0x0)/Pci(0x1A,0x0)/Pci(0x0,0x0)/NVMe(0x1,B8-77-91-31-53-38-25-00)/HD(1,GPT,0D5E68B3-7832-443B-8F72-266704FF0820,0x22,0x7FDE)
     BLK7: Alias(s):
          PciRoot(0x0)/Pci(0x1D,0x0)/Pci(0x0,0x0)/NVMe(0x1,45-D8-DB-2A-01-75-A0-00)

@cryptearth: Fully agree with what you say. Using a different fs and a driver from efifs is something I didn't think of, thanks for the suggestion. I keep this in mind when I experiment further. Also agree re: consumer vs pro hardware; my Supermicro boards didn't give me that much headache in years than what this one gave me just in a few weeks.

But this board is what I have in hand right now. I certainly try your suggestion later on since I am curious, but for the moment, I have settled on tried-and-true GRUB which works flawless for this situation.

I would extend your market share / testing argument to the systemd-boot mechanism. It's no exaggeration to say that the GRUB user base is at least hundred times bigger than the systemd-boot user base. Look at the UEFI map above, this system has quite a lot of devices at different places which could trigger bugs. Also, systemd-boot and especially the Discoverable Partitions mechanism was meant for automated deployments; I am not sure if it's sufficiently tested on systems with more than one drive, all different controllers (SATA/NVMe) etc. Thus my switch to GRUB.

Thanks for all the help I got. The suggestions helped to shape my assessment of the situation and avoid wasting even more time.

TL;DR for future readers in a similar situation: Cut your losses and switch to GRUB, this is better tested and has higher chances to work

Last edited by emk2203 (2024-10-18 08:58:31)

cryptearth · 2024-10-18 10:11:51

I gave several different types of booting a try in a vm as I wanted to figure if one is "better" (or at least more convenient) than others - but couldn't find any as "best" without also comming with its own fair share of issues
I got best results from grub - but I think this is mostly due to I have the most experience with it while still struggle with others like systemd

on the other side I also had one test setup with systemd with auto discover and no fstab at all - and at least within the vm it worked without issues (although not tested on bare metal)

as for your issue: I'm currently on phone so it's a bit hard to understand the mapping - but as the shell has a FS mapping for the additional xbootldr partition it seems that at least the shell sees it and can use it - which should be inhired from the uefi and hence should be the case for systemd, too
so, aside from the uefi firmware being the issue here it also could be the combination of both systemd and the firmware causing bugs - although I do believe systemd being very robust

anyway - its yet another of those "according to specs it should work - but due to some weird quirks and bugs it just doesn't"
recently there's a topic about an acer laptop which for sone reason locked the OP out of accessing the uefi because of an additional boot-entry - and only after it was cleared OP could access the settings again
the only reply from acer support was: "we don't support linux" - they completely missed the point it's thier crap firmware

cryptearth · 2024-10-19 12:03:51

btw - I looked up uefi spec: section 13 doesn't help here - seems an implementation bug

Arch Linux

#1 2024-10-17 17:40:23

Cannot boot Arch from systemd-boot installation with XBOOTLDR

#2 2024-10-17 17:52:21

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#3 2024-10-17 18:37:51

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#4 2024-10-17 18:39:13

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#5 2024-10-17 21:04:10

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#6 2024-10-18 06:37:56

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#7 2024-10-18 08:56:37

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#8 2024-10-18 10:11:51

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

#9 2024-10-19 12:03:51

Re: Cannot boot Arch from systemd-boot installation with XBOOTLDR

Board footer