Early boot freeze on newer kernels

gromit · 2024-07-07 18:57:22

Please test the following:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-lts-v6.6.17.r315.gfa24408-1-x86_64.pkg.tar.zst

mavrix · 2024-07-07 19:04:43

Still good.

gromit · 2024-07-07 19:44:44

Seems like https://git.kernel.org/pub/scm/linux/ke … 8b6171f5fe is the first bad commit:

commit 8117961d98fb2d335ab6de2cad7afb8b6171f5fe
Author: Ard Biesheuvel <ardb@kernel.org>
Date:   Tue Sep 12 09:00:53 2023 +0000

    x86/efi: Disregard setup header of loaded image
    
    commit 7e50262229faad0c7b8c54477cd1c883f31cc4a7 upstream.
    
    The native EFI entrypoint does not take a struct boot_params from the
    loader, but instead, it constructs one from scratch, using the setup
    header data placed at the start of the image.
    
    This setup header is placed in a way that permits legacy loaders to
    manipulate the contents (i.e., to pass the kernel command line or the
    address and size of an initial ramdisk), but EFI boot does not use it in
    that way - it only copies the contents that were placed there at build
    time, but EFI loaders will not (and should not) manipulate the setup
    header to configure the boot. (Commit 63bf28ceb3ebbe76 "efi: x86: Wipe
    setup_data on pure EFI boot" deals with some of the fallout of using
    setup_data in a way that breaks EFI boot.)
    
    Given that none of the non-zero values that are copied from the setup
    header into the EFI stub's struct boot_params are relevant to the boot
    now that the EFI stub no longer enters via the legacy decompressor, the
    copy can be omitted altogether.
    
    Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
    Signed-off-by: Ingo Molnar <mingo@kernel.org>
    Link: https://lore.kernel.org/r/20230912090051.4014114-19-ardb@google.com
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

 drivers/firmware/efi/libstub/x86-stub.c | 46 ++++++----------------------------------------
 1 file changed, 6 insertions(+), 40 deletions(-)

According to kernel.dance there is a fix for this in v6.6.24 "x86/efistub: Reinstate soft limit for initrd loading", does that work?

mavrix · 2024-07-07 20:16:34

Wouldn't that still be included in the newer kernels then? Those don't work. Neither does

sudo pacman -U https://archive.archlinux.org/packages/l/linux-lts/linux-lts-6.6.24-1-x86_64.pkg.tar.zs

I just tried.

gromit · 2024-07-07 22:05:32

Did you already try the latest mainline as I asked in https://bbs.archlinux.org/viewtopic.php … 4#p2182214? Maybe I have missed it but I don't think you have ever answered to that

Maybe we can gather some more information and write a bugreport to the kernel devs ... Maybe the thread for the fixing commit is a good start to see what debug outputs they want: https://lore.kernel.org/all/a99a831a-8a … dgorny.cz/

Last edited by gromit (2024-07-07 22:06:23)

loqs · 2024-07-07 22:15:10

Does linux-lts-6.6.37-1.1-x86_64.pkg.tar.zst/linux-lts-headers-6.6.37-1.1-x86_64.pkg.tar.zst which has 8117961d98fb2d335ab6de2cad7afb8b6171f5fe reverted boot?

Last edited by loqs (2024-07-07 22:16:19)

mavrix · 2024-07-08 17:37:02

mavrix wrote:

Mainline freezes as well btw.

6.6.37 with the reverted commit boots fine otoh.

$ uname -r
6.6.37-1.1-lts

gromit · 2024-07-08 19:05:43

So to me this looks like the fixing commit is missing your hardware and its the same underlying bug, what do you think @loqs?

Did you already update your BIOS/UEFI just for good measure?

mavrix · 2024-07-08 19:41:57

The hardware's a bit dated by now, so there haven't been any bios updates in a while. It's on the most recent version, though. systemd-boot is up to date as is the rest of my system - I do full system upgrades frequently and run systemd-boot-update regularly as well. Running it again only results in

Starting Automatic Boot Loader Update...
Skipping "/boot/EFI/systemd/systemd-bootx64.efi", same boot loader version in place already.
Skipping "/boot/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Skipping "/boot/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Finished Automatic Boot Loader Update.

loqs · 2024-07-08 19:51:27

In case upstream wants to check against mainline; linux-mainline 6.10-rc7 with 7e50262229faad0c7b8c54477cd1c883f31cc4a7 reverted linux-mainline-6.10rc7-1.1-x86_64.pkg.tar.zst/linux-mainline-headers-6.10rc7-1.1-x86_64.pkg.tar.zst. 5f51c5d0e905608ba7be126737f7c84a793ae1aa does not revert cleanly from 6.10-rc7 do that can not be tested.

@gromit does look very much like a hardware/firmware specific issue.

@marvix can you post the outputs from the diagnostics upstream wanted for the other issue related to the same commit:

# dmidecode
# bootctl
# dmesg # from a working boot

mavrix · 2024-07-08 21:10:11

Ok. Here goes. dmidecode and dmesg are a bit long, so I linked them:

dmidecode - https://drive.proton.me/urls/GXTZVVSTK0#43CCvNy1m6dh
dmesg - https://drive.proton.me/urls/6DSXK4R2D8#8TFGkw6yvFfx

# bootctl
System:
      Firmware: UEFI 2.40 (American Megatrends 5.11)
 Firmware Arch: x64
   Secure Boot: disabled
  TPM2 Support: no
  Measured UKI: no
  Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 256.1-1-arch-g34ba18b^
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Enroll SecureBoot keys
               ✓ Retain SHIM protocols
               ✓ Menu can be disabled
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/dcb2ccc2-4472-46df-9594-009416849e4c
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 System Token: set
       Exists: yes

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/dcb2ccc2-4472-46df-9594-009416849e4c)
         File: ├─/EFI/systemd/PreLoader.efi
               ├─/EFI/systemd/systemd-bootx64.efi (systemd-boot 256.1-1-arch-g34ba18b^)
               ├─/EFI/systemd/HashTool.efi
               ├─/EFI/systemd/loader.efi (systemd-boot 229)
               └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 256.1-1-arch-g34ba18b^)

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/dcb2ccc2-4472-46df-9594-009416849e4c
         File: └─/EFI/systemd/systemd-bootx64.efi

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/dcb2ccc2-4472-46df-9594-009416849e4c)
        token: arch

Default Boot Loader Entry:
         type: Boot Loader Specification Type #1 (.conf)
        title: Arch Linux
           id: arch_default.conf
       source: /boot//loader/entries/arch_default.conf
        linux: /boot//vmlinuz-linux
       initrd: /boot//initramfs-linux.img
      options: rd.luks.uuid=dd56e8e2-8a1c-4691-9566-c1d433d1311f rw rd.lvm.vg=archvgxps root=UUID=3c3f46b4-fe6a-425f-8f7f-53e45b731eea resume=UUID=1edcf870-ce37-4799-ac32-c2457dde344d rd.luks.options=discard

gromit · 2024-07-08 21:29:18

Do you want to write the bugreport to the kernel devs or should I do it @mavrix?

mavrix · 2024-07-09 18:25:02

Might as well give it a shot myself. Any idea what the most sensible way would be to go about this? Simply reply to the e-mail thread you mentioned @gromit? Or would that be considered necrobumping and a fresh report the classier option with maybe a reference to the earlier one? Would it be in order for me to CC you as well @gromit?

gromit · 2024-07-09 18:37:07

(I'll just dump some thoughts regarding the bug report)

I think replying to the initial report is fine, but I am not sure myself. Did you already see that you can use the "mailto:" link on the Lore interface to hook into the thread?
Feel free to CC me!

It also can't hurt to mention this forum thread, but making the bug report as standalone as possible is a good idea since some devs don't like looking at external resources.

Also try to include as much Information as possible (especially the information they wanted from the previous reporter) and to which commit we have bisected the issue.
You should also mention that we have tried the latest mainline and that reverting the commit in question fixes the issue but the fix does not help your case.

If you want to be nice you can add a "Bisected-by: Christian Heusel <christian@heusel.eu>" to honor honor my efforts with the kernel building

loqs · 2024-07-17 18:40:50

linux 6.10 with the partial revert suggested in https://lore.kernel.org/regressions/CAM … gmail.com/ applied linux-mainline-6.10-1.1-x86_64.pkg.tar.zst/linux-mainline-headers-6.10-1.1-x86_64.pkg.tar.zst

mavrix · 2024-07-17 22:07:14

Thanks. The kernel with the partial revert won't boot either. I replied accordingly in the bug report thread.

loqs · 2024-07-17 22:55:59

linux 6.10 with 2-0001-Partial-revert-of-8117961d98f-2.patch applied linux-mainline-6.10-1.2-x86_64.pkg.tar.zst/linux-mainline-headers-6.10-1.2-x86_64.pkg.tar.zst.
Starting on the next build.

mavrix · 2024-07-17 23:22:25

No luck with this one.

loqs · 2024-07-17 23:32:19

linux 6.10 with 3-0001-Partial-revert-of-8117961d98f-1.patch applied linux-mainline-6.10-1.3-x86_64.pkg.tar.zst/linux-mainline-headers-6.10-1.3-x86_64.pkg.tar.zst.

mavrix · 2024-07-18 20:23:34

This one works.

loqs · 2024-07-18 22:21:44

Anticipating the next request, linux 6.10 with 2-0001-Dump-setup-header.patch from https://lore.kernel.org/all/CAMj1kXF5TU … gmail.com/ applied linux-mainline-6.10-1.4-x86_64.pkg.tar.zst/linux-mainline-headers-6.10-1.4-x86_64.pkg.tar.zst.
Edit:
linux 6.10 with 2-0001-Dump-setup-header.patch + 3-0002-Set-some-header-flags.patch from https://lore.kernel.org/all/CAMj1kXF5TU … gmail.com/ applied linux-mainline-6.10-1.5-x86_64.pkg.tar.zst/linux-mainline-headers-6.10-1.5-x86_64.pkg.tar.zst.

Last edited by loqs (2024-07-18 22:55:12)

mavrix · 2024-08-08 17:57:46

Finally back up and running on Arch Stable again.

$ uname -r
6.10.3-arch1-2

Thank you @gromit and @loqs for all the help with troubleshooting and reporting.
Cheers!

Arch Linux

#26 2024-07-07 18:57:22

Re: Early boot freeze on newer kernels

#27 2024-07-07 19:04:43

Re: Early boot freeze on newer kernels

#28 2024-07-07 19:44:44

Re: Early boot freeze on newer kernels

#29 2024-07-07 20:16:34

Re: Early boot freeze on newer kernels

#30 2024-07-07 22:05:32

Re: Early boot freeze on newer kernels

#31 2024-07-07 22:15:10

Re: Early boot freeze on newer kernels

#32 2024-07-08 17:37:02

Re: Early boot freeze on newer kernels

#33 2024-07-08 19:05:43

Re: Early boot freeze on newer kernels

#34 2024-07-08 19:41:57

Re: Early boot freeze on newer kernels

#35 2024-07-08 19:51:27

Re: Early boot freeze on newer kernels

#36 2024-07-08 21:10:11

Re: Early boot freeze on newer kernels

#37 2024-07-08 21:29:18

Re: Early boot freeze on newer kernels

#38 2024-07-09 18:25:02

Re: Early boot freeze on newer kernels

#39 2024-07-09 18:37:07

Re: Early boot freeze on newer kernels

#40 2024-07-17 18:40:50

Re: Early boot freeze on newer kernels

#41 2024-07-17 22:07:14

Re: Early boot freeze on newer kernels

#42 2024-07-17 22:55:59

Re: Early boot freeze on newer kernels

#43 2024-07-17 23:22:25

Re: Early boot freeze on newer kernels

#44 2024-07-17 23:32:19

Re: Early boot freeze on newer kernels

#45 2024-07-18 20:23:34

Re: Early boot freeze on newer kernels

#46 2024-07-18 22:21:44

Re: Early boot freeze on newer kernels

#47 2024-08-08 17:57:46

Re: Early boot freeze on newer kernels

Board footer