[SOLVED] Grub update causes delay in loading ramdisk

Tigermouthbear · 2022-08-30 00:03:15

I also got this problem on my fresh install of Arch with EFI, root partition LUKS + LVM2 encrypted, and an unencrypted boot partition. I was about to shrug the slow booting off as something I did wrong, when I found this thread. Downgrading GRUB to 2.06.r261 fixed the problem and saved me about 23s at boot.

Anyways, I created a forum account just to say thanks! and confirm the problem was happening to other people too.

lenhuppe · 2022-08-30 01:03:16

The solution for me was to change the way that I am setting up my zpool & zfs cache following the instructions here: [https://openzfs.github.io/openzfs-docs/ … inux/index]

I switched from using zfs-import-cache to using zfs-import-scan, and I enabled grub2 compatibility on my zpool.

As I write this my test system is booting from Grub r322 with no delays

Last edited by lenhuppe (2022-08-30 11:28:47)

lenhuppe · 2022-08-30 01:19:31

Tigermouthbear wrote:

I also got this problem on my fresh install of Arch with EFI, root partition LUKS + LVM2 encrypted, and an unencrypted boot partition. I was about to shrug the slow booting off as something I did wrong, when I found this thread. Downgrading GRUB to 2.06.r261 fixed the problem and saved me about 23s at boot.
Anyways, I created a forum account just to say thanks! and confirm the problem was happening to other people too.

If Grub r261 is working you have the option of staying with it until you are ready to rebuild your zpool.

Your use case is very different than mine so you may need to tinker a bit on a test system like I did.

It took me ten days of testing to find the answer.

LanMarc77 · 2022-08-30 11:26:56

Tigermouthbear wrote:

I also got this problem on my fresh install of Arch with EFI, root partition LUKS + LVM2 encrypted, and an unencrypted boot partition. I was about to shrug the slow booting off as something I did wrong, when I found this thread. Downgrading GRUB to 2.06.r261 fixed the problem and saved me about 23s at boot.
Anyways, I created a forum account just to say thanks! and confirm the problem was happening to other people too.

Welcome to the forum.
Which root fs do you use? ZFS? And which bios manufacturer (AMI, Phoenix...)?
I am trying to find similarities and differences.
Lenhuppe are you able to test if the r297 also works fast with your settings, avoiding the delay? I am not using btrfs and another person uses ext4. Any idea what could be changed for these filesystems to not trigger the bug?

lenhuppe · 2022-08-30 11:38:56

LanMarc77 wrote:

Lenhuppe are you able to test if the r297 also works fast with your settings, avoiding the delay?

I can run that test when I am home from work which will be late tonight my time.

Last edited by lenhuppe (2022-08-30 12:23:02)

lenhuppe · 2022-08-30 12:25:35

LanMarc77 wrote:

Any idea what could be changed for these filesystems to not trigger the bug?

What I think needs to change is the way that zfs is configured. A grub2 compatibility mode has been added which helps, but Grub only supports a subset of what OpenZFS has to offer. That is why the OpenZFS wiki has you create a separate boot pool. That approach allows you to configure your data pools any way you want while maintaining Grub compatibility in your boot pool. My use case is simple and a fully grub compatible zfs pool works fine for me.

Tigermouthbear · 2022-08-30 15:12:21

LanMarc77 wrote:

Welcome to the forum.
Which root fs do you use? ZFS? And which bios manufacturer (AMI, Phoenix...)?
I am trying to find similarities and differences.
Lenhuppe are you able to test if the r297 also works fast with your settings, avoiding the delay? I am not using btrfs and another person uses ext4. Any idea what could be changed for these filesystems to not trigger the bug?

I am using ext4 on my root partition and my bios is Asus version G531GT.306

The day before I got this problem on my Asus laptop, I installed Arch w/ grub on an old Dell Latitude laptop (ext4 root + LUKS/LVM2 encryption). I had no problems with that booting using the latest grub version, and the big differences during installation were that the Dell used MBR and had a HDD, while the Asus used EFI and had a SSD.

So while I was trying to fix the problem, I thought it was a problem with grub booting from EFI into a luks root or a problem with my SSD.

lenhuppe · 2022-08-31 02:30:01

LanMarc77 wrote:

Lenhuppe are you able to test if the r297 also works fast with your settings, avoiding the delay?

Today I was able to boot from Grub r261, r297 and r322 with no delays using my new zpool configuration.

Last edited by lenhuppe (2022-08-31 18:01:35)

LanMarc77 · 2022-08-31 18:52:42

Ok, as I now have a test system I will try myself on that bisecting process to finally get the problematic commit.

lenhuppe · 2022-09-01 00:18:47

LanMarc77 wrote:

Ok, as I now have a test system I will try myself on that bisecting process to finally get the problematic commit.

I am interested in knowing if there are any changes which would negatively impact the zfs-import-cache.service.

LanMarc77 · 2022-09-01 09:40:07

lenhuppe wrote:

LanMarc77 wrote:
Ok, as I now have a test system I will try myself on that bisecting process to finally get the problematic commit.
I am interested in knowing if there are any changes which would negatively impact the zfs-import-cache.service.

I found the responsible commit area https://bugs.archlinux.org/task/75673#comment210768.
This diff seems to mainly deal with memory management so I can not tell if the zfs-import-cache.service is impacted. But it might very well be that different preconditions including layout and configurations of filesystems lead to different input parameters of the changed function triggering the longer delay. Not everyone seems to experience the delay.
I will signup for the grub devel list and report the situation there.

lenhuppe · 2022-09-01 14:21:10

LanMarc77 wrote:

I found the responsible commit area https://bugs.archlinux.org/task/75673#comment210768.
This diff seems to mainly deal with memory management so I can not tell if the zfs-import-cache.service is impacted. But it might very well be that different preconditions including layout and configurations of filesystems lead to different input parameters of the changed function triggering the longer delay. Not everyone seems to experience the delay.
I will signup for the grub devel list and report the situation there.

I have been looking at various configuration files and there is definitely a large number of settings. In addition I have been reading forum posts and It looks to me like there is no consensus on how to use the zfs cache. If you look at the zfs-import-cache and zfs-import-scan service files they are very different. Its clear to me that they are intended for different use cases. I think that you are right about preconditions and filesystem setups having a large impact.

I need to do more testing but it looks to me like the 'zpool set bootfs=<pool>/<root dataset> <pool>' setting and the zfs-import-scan service work well together. In the past I had been manually exporting and then re-importing my pool before enabling the zfs-import-cache service. That configuration still works but not very well. Whether that is a design decision or a bug I cannot say.

That said thank you for doing the bisect ... I need to learn how to do that.

uka · 2022-09-01 18:17:30

Can confirm the issue in r322 but downgraded to r297 and it is gone. Thanks LanMarc77 for letting upstream know, I read your entry at the mailing list. In my case I boot to a non-encrypted btrfs root partition from a regular boot/EFI partition. Nothing exotic. This was in an AMI BIOS/Intel system, while an AMI BIOS/AMD system with the same plain vanilla filesystem configuration didn't have longer boot times problems at all.

Last edited by uka (2022-09-01 19:21:05)

LanMarc77 · 2022-09-02 20:02:07

eworm has patched the grub package under current assumptions that might fix the issue for most. It is currently available in testing https://archlinux.org/packages/testing/x86_64/grub/
This patch works for me. Can someone else please try and test and report back here?
Lenhuppe, as you do have worked your way around this differently I will not ask of you to go back the long way unless you can do this easily to verify this patch works with your original configuration.

lenhuppe · 2022-09-02 23:50:20

LanMarc77 wrote:

eworm has patched the grub package under current assumptions that might fix the issue for most. It is currently available in testing https://archlinux.org/packages/testing/x86_64/grub/
This patch works for me. Can someone else please try and test and report back here?
Lenhuppe, as you do have worked your way around this differently I will not ask of you to go back the long way unless you can do this easily to verify this patch works with your original configuration.

My test setup worked equally well in two configurations. My original setup which used the zfs-import-cache and my new setup which uses zfs-import-scan and has the zpool cache disabled.

In both cases my setup is a simple one with only one pool and no encryption. I have not yet been able to setup a separate boot and root pool but I am working on it.

uka · 2022-09-04 22:03:58

LanMarc77 wrote:

eworm has patched the grub package under current assumptions that might fix the issue for most. It is currently available in testing https://archlinux.org/packages/testing/x86_64/grub/
This patch works for me. Can someone else please try and test and report back here?
Lenhuppe, as you do have worked your way around this differently I will not ask of you to go back the long way unless you can do this easily to verify this patch works with your original configuration.

Can confirm that grub 2:2.06.r322.gd9b4638c5-4 doesn't have the delay, however I wasn't paying attention that the package for grub was coming from development/git branches instead of the last stable version. I don't think it is a good idea to have a key component packaged from development sources, so for my own sanity after testing this patch, I downgraded to stable grub 2.06 (arch build 5) from the archives to https://archive.archlinux.org/packages/ … kg.tar.zst. It works as expected, will keep it for a while.

lenhuppe · 2022-09-05 01:41:42

uka wrote:

LanMarc77 wrote:
eworm has patched the grub package under current assumptions that might fix the issue for most. It is currently available in testing https://archlinux.org/packages/testing/x86_64/grub/
This patch works for me. Can someone else please try and test and report back here?
Lenhuppe, as you do have worked your way around this differently I will not ask of you to go back the long way unless you can do this easily to verify this patch works with your original configuration.
Can confirm that grub 2:2.06.r322.gd9b4638c5-4 doesn't have the delay, however I wasn't paying attention that the package for grub was coming from development/git branches instead of the last stable version. I don't think it is a good idea to have a key component packaged from development sources, so for my own sanity after testing this patch, I downgraded to stable grub 2.06 (arch build 5) from the archives to https://archive.archlinux.org/packages/ … kg.tar.zst. It works as expected, will keep it for a while.

Can appreciate that some users need to be cautious. I now have two systems running Grub 2:2.06.r322.gd9b4638c5-4 and I can confirm that they are booting with no known issues. If I do take notice of any issues I will post here.

That said my systems are booting faster than ever now that I have reconfigured my zfs setup. Most notably I stopped using the zfs cache. The OpenZFS wiki makes no mention of it and the following excerpt from /etc/default/zfs corroborates that:

# Generally, the use of a cache file is usually not recommended on Linux
# because it sometimes is more trouble than it's worth (laptops with external
# devices or when/if device nodes changes names).

Happy trails !

Last edited by lenhuppe (2022-09-05 19:16:14)

nekys · 2022-09-05 18:41:09

I'm in day 5 dealing with loaders' delay (systemd-analyze) and i'm glad to find this post a few minutes ago.
@lenhuppe thanks for your effort tracking down this and big thanks to @uka providing a downgrade package for grub.
Since then, loader time drops down by ~4.500ms and i'm back to normal.
I need to mention that i have a pretty simple partition setup with one nvme and two separate partitions. One for / and the other for /boot.

lenhuppe · 2022-09-05 18:57:27

nekys wrote:

I'm in day 5 dealing with loaders' delay (systemd-analyze) and i'm glad to find this post a few minutes ago.
@lenhuppe thanks for your effort tracking down this and big thanks to @uka providing a downgrade package for grub.
Since then, loader time drops down by ~4.500ms and i'm back to normal.
I need to mention that i have a pretty simple partition setup with one nvme and two separate partitions. One for / and the other for /boot.

Any time that you feel the need to backtrack on a package you can check the archives here: https://archive.archlinux.org/

Arch Linux

#26 2022-08-30 00:03:15

Re: [SOLVED] Grub update causes delay in loading ramdisk

#27 2022-08-30 01:03:16

Re: [SOLVED] Grub update causes delay in loading ramdisk

#28 2022-08-30 01:19:31

Re: [SOLVED] Grub update causes delay in loading ramdisk

#29 2022-08-30 11:26:56

Re: [SOLVED] Grub update causes delay in loading ramdisk

#30 2022-08-30 11:38:56

Re: [SOLVED] Grub update causes delay in loading ramdisk

#31 2022-08-30 12:25:35

Re: [SOLVED] Grub update causes delay in loading ramdisk

#32 2022-08-30 15:12:21

Re: [SOLVED] Grub update causes delay in loading ramdisk

#33 2022-08-31 02:30:01

Re: [SOLVED] Grub update causes delay in loading ramdisk

#34 2022-08-31 18:52:42

Re: [SOLVED] Grub update causes delay in loading ramdisk

#35 2022-09-01 00:18:47

Re: [SOLVED] Grub update causes delay in loading ramdisk

#36 2022-09-01 09:40:07

Re: [SOLVED] Grub update causes delay in loading ramdisk

#37 2022-09-01 14:21:10

Re: [SOLVED] Grub update causes delay in loading ramdisk

#38 2022-09-01 18:17:30

Re: [SOLVED] Grub update causes delay in loading ramdisk

#39 2022-09-02 20:02:07

Re: [SOLVED] Grub update causes delay in loading ramdisk

#40 2022-09-02 23:50:20

Re: [SOLVED] Grub update causes delay in loading ramdisk

#41 2022-09-04 22:03:58

Re: [SOLVED] Grub update causes delay in loading ramdisk

#42 2022-09-05 01:41:42

Re: [SOLVED] Grub update causes delay in loading ramdisk

#43 2022-09-05 18:41:09

Re: [SOLVED] Grub update causes delay in loading ramdisk

#44 2022-09-05 18:57:27

Re: [SOLVED] Grub update causes delay in loading ramdisk

Board footer