[SOLVED] Grub update causes delay in loading ramdisk

lenhuppe · 2022-08-19 14:36:38

Today I upgraded Grub from 2.06.r261 to 2.06.r297 and I now have a long delay when loading the initial ramdisk:

Booting 'Arch Linux'
Loading Linux linux-lts ...
Loading initial ramdisk ...

I noticed after upgrading that there is no pacman hook to install and/or recreate the grub config. I am guessing that there is a reason for this. Therefore I did a manual install and config as follows:

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

The zpool parameter is there because I am using OpenZFS. I tried enabling debugging but it has not given me any usable information yet.

As I write this I have been waiting for 5 minutes for grub to load with debugging enabled.

Last edited by lenhuppe (2022-08-30 01:03:51)

Scimmia · 2022-08-19 14:41:39

Since updating the grub package doesn't update the actual installed bootloader, I'd say you're probably looking in the wrong place.

Edit: wait, you did the grub-install before rebooting? I was thinking you were saying you did that to try to fix it.

Last edited by Scimmia (2022-08-19 14:59:36)

lenhuppe · 2022-08-19 15:07:00

Scimmia wrote:

Since updating the grub package doesn't update the actual installed bootloader, I'd say you're probably looking in the wrong place.
Edit: wait, you did the grub-install before rebooting? I was thinking you were saying you did that to try to fix it.

I did a manual grub-install followed by a grub-mkconfig before rebooting.

Aside from the ramdisk issue the upgrade appears to have succeeded.

I enabled debug mode to see if that would tell me anything useful but it did not.

Last edited by lenhuppe (2022-08-19 15:09:42)

twelveeighty · 2022-08-19 19:42:19

Isn't this the same (unsolved) issue as this: https://bbs.archlinux.org/viewtopic.php?id=278895

LanMarc77 · 2022-08-19 21:24:36

Hi,
short term (3 months) arch user with no issues so far.
I basically did the same as lenhuppe.
Today, after I was running the new 5.19 kernel for a few days. Some people seem to have trouble when having encrypted lvm volumes. I also have an encrypted root and boot, but plain partitions without lvm.

I also noticed that after (re)installing=updating grub with grub-install followed by grub-mkconfig, that both of my boot up stages, first decrypting of /boot and then booting after selecting a menu entry takes much longer.
Not 5 minutes though. It increased by around 15s for each stage.

I suspect some changes in the defaults that grub is now using causing the additional delays (e.g. searching the system for disks)
It doesn't look to me as it is related to the issue twelveeighty linked.
As I was already running the 5.19.2 kernel before the grub update, I also do not think it is related to the new kernel.

lenhuppe · 2022-08-20 01:19:50

twelveeighty wrote:

Isn't this the same (unsolved) issue as this: https://bbs.archlinux.org/viewtopic.php?id=278895

This is not really the same issue. My system is booting ok, it just takes noticeably longer after the update.

I have since installed a new Arch system onto my test machine and it does boot, it just takes a long time.

lenhuppe · 2022-08-21 18:41:21

LanMarc77 wrote:

Hi,
short term (3 months) arch user with no issues so far.
I basically did the same as lenhuppe.
Today, after I was running the new 5.19 kernel for a few days. Some people seem to have trouble when having encrypted lvm volumes. I also have an encrypted root and boot, but plain partitions without lvm.
I also noticed that after (re)installing=updating grub with grub-install followed by grub-mkconfig, that both of my boot up stages, first decrypting of /boot and then booting after selecting a menu entry takes much longer.
Not 5 minutes though. It increased by around 15s for each stage.
I suspect some changes in the defaults that grub is now using causing the additional delays (e.g. searching the system for disks)
It doesn't look to me as it is related to the issue twelveeighty linked.
As I was already running the 5.19.2 kernel before the grub update, I also do not think it is related to the new kernel.

Reverting to Grub 2.06.r261 fixed my problem.

Have you given that a try?

LanMarc77 · 2022-08-21 20:32:31

lenhuppe wrote:

Reverting to Grub 2.06.r261 fixed my problem.
Have you given that a try?

No. I am not annoyed enough by it (yet). But good to know this would work. Also that it seems a brand new systems seem to have the same issue. It would be interesting to test if the new archiso has the same issue when a new snapshot is made assuming it has that new version then.
All the tutorials one finds when searching upgrade grub usually do not say anything about that one also needs to update the binaries and not just the config. So I guess not many have discovered the additional delay yet.
I had taken a look at the diffs from the grub repo but could not find something obvious that would explain the additional delay. But I understood only half of what I read...

Do you think it makes sense to file a bug report in the official Bugtracker?

lenhuppe · 2022-08-21 23:33:52

LanMarc77 wrote:

lenhuppe wrote:
Reverting to Grub 2.06.r261 fixed my problem.
Have you given that a try?
No. I am not annoyed enough by it (yet). But good to know this would work. Also that it seems a brand new systems seem to have the same issue. It would be interesting to test if the new archiso has the same issue when a new snapshot is made assuming it has that new version then.
All the tutorials one finds when searching upgrade grub usually do not say anything about that one also needs to update the binaries and not just the config. So I guess not many have discovered the additional delay yet.
I had taken a look at the diffs from the grub repo but could not find something obvious that would explain the additional delay. But I understood only half of what I read...
Do you think it makes sense to file a bug report in the official Bugtracker?

Yes I think that a bug report is called for. I am not sure how to go about doing that. That last time that I tried I worded it wrong and it did not go well.

Last edited by lenhuppe (2022-08-22 00:01:31)

lenhuppe · 2022-08-22 17:28:27

LanMarc77 wrote:

It would be interesting to test if the new archiso has the same issue when a new snapshot is made assuming it has that new version then.

I built an ISO using archiso version 65-1 and the Grub version is 2.06.r261. This was after reverting so I am guessing that the mkarchiso script uses the grub package already installed on the system.

Last edited by lenhuppe (2022-08-22 20:01:42)

lenhuppe · 2022-08-22 17:41:54

Since the latest version of Grub is not a necessity I modified my installer to use version 2.06.r261 from the archives.

If you find the answer please share.

Last edited by lenhuppe (2022-08-22 17:42:27)

LanMarc77 · 2022-08-22 18:17:30

Hi,
thanks for your detective work. I opened a bug report https://bugs.archlinux.org/task/75673 with all our findings referencing this topic.

How do I "fix" a package with pacman so that it uses the older version. I am coming from debian where I can set packages onhold so they are not updated.

twelveeighty · 2022-08-22 18:54:49

LanMarc77 wrote:

How do I "fix" a package with pacman so that it uses the older version.

man pacman, search for --ignore

lenhuppe · 2022-08-22 19:21:55

LanMarc77 wrote:

Hi,
thanks for your detective work. I opened a bug report https://bugs.archlinux.org/task/75673 with all our findings referencing this topic.
How do I "fix" a package with pacman so that it uses the older version. I am coming from debian where I can set packages onhold so they are not updated.

Thank you for handling the bug report, and for the kind words. I am not a programmer but I am a master troubleshooter.

Fortunately pacman does allow you to place a hold on upgrades. Here is my quick and dirty solution:

#!/usr/bin/env bash

# install grub package
pacman -U https://archive.archlinux.org/packages/g/grub/grub-2%3A2.06.r261.g2f4430cc0-1-x86_64.pkg.tar.zst

# disable grub updates
sed -i '{
	s/#IgnorePkg.*/IgnorePkg =/	
	s/^IgnorePkg.*/& grub/
} ' /etc/pacman.conf

# install grub to disk
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB

# generate grub config
grub-mkconfig -o /boot/grub/grub.cfg

If I check for updates I get the following:

pacman -Syu

:: Synchronizing package databases...
 core downloading...
 extra downloading...
 community downloading...
:: Starting full system upgrade...
warning: grub: ignoring package upgrade (2:2.06.r261.g2f4430cc0-1 => 2:2.06.r297.g0c6c1aff2-1)
 there is nothing to do

Last edited by lenhuppe (2022-08-27 03:53:38)

LanMarc77 · 2022-08-23 19:56:20

Thank you all for your suggestions. As always, more than one way to achieve something.
We will wait for what happens with "our" bug.

LanMarc77 · 2022-08-24 18:58:24

Lenhuppe, would you be able to give this https://bugs.archlinux.org/task/75673 a test ride?
I understood you have a test system available and I only have one system myself at the moment and a bit hesitant to try it on my only system that I need.

lenhuppe · 2022-08-24 20:23:35

LanMarc77 wrote:

Lenhuppe, would you be able to give this https://bugs.archlinux.org/task/75673 a test ride?
I understood you have a test system available and I only have one system myself at the moment and a bit hesitant to try it on my only system that I need.

Yes I can, I will do a test run later this evening when I am home from work.

In the meantime I am looking into how to bisect packages. Any advice is welcome.

Last edited by lenhuppe (2022-08-24 22:38:00)

LanMarc77 · 2022-08-24 20:36:12

Great.
I understood eworm seems to take over this work and only needs us to tell him if the build he pushes to testing is working or not.
He pushed grub 2:2.06.r322.gd9b4638c5-1 already.

lenhuppe · 2022-08-24 22:38:15

LanMarc77 wrote:

Great.
I understood eworm seems to take over this work and only needs us to tell him if the build he pushes to testing is working or not.
He pushed grub 2:2.06.r322.gd9b4638c5-1 already.

Installed grub 2:2.06.r322 from testing repo with no issues. To answer eworm's question, I am running the linux-lts kernel on OpenZFS 2.1.5.

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

It loads the kernel faster (Loading Linux linux-lts ...) but it still has a long delay when loading the ramdisk (Loading initial ramdisk ...)

Last edited by lenhuppe (2022-08-24 22:40:24)

lenhuppe · 2022-08-24 23:55:38

LanMarc77 wrote:

Great.
I understood eworm seems to take over this work and only needs us to tell him if the build he pushes to testing is working or not.
He pushed grub 2:2.06.r322.gd9b4638c5-1 already.

I used mkarchiso to create a custom ISO because Grub is a dep when installing archiso. Since grub r322 is already installed mkarchiso used that. When the ISO booted I verified Grub r322 and I did not see any long delay.

The difference is that my custom ISO does not boot off of OpenZFS. The issue may be with support for OpenZFS. I use the "ZPOOL_VDEV_NAME_PATH=1" parameter because w/out it grub-mkconfig will fail on OpenZFS.

Last edited by lenhuppe (2022-08-25 00:09:27)

lenhuppe · 2022-08-25 01:40:48

LanMarc77 wrote:

Great.
I understood eworm seems to take over this work and only needs us to tell him if the build he pushes to testing is working or not.
He pushed grub 2:2.06.r322.gd9b4638c5-1 already.

There is a bug with OpenZFS which requires the environmental variable "ZPOOL_VDEV_NAME_PATH=1" to be set. I tried editing the grub config file /etc/default/grub before running grub-mkconfig as follows:

GRUB_CMDLINE_LINUX="ZPOOL_VDEV_NAME_PATH=1"

With grub r297 the kernel loaded a little slower but the ramdisk loaded quickly.

With grub v322 both the kernel and ramdisk took a long time to load.

Maybe I am not inserting the parameter in the right place?

Last edited by lenhuppe (2022-08-25 01:42:47)

torvic9 · 2022-08-26 07:35:16

Hi,

Since the newest GRUB r322 completely borked the boot, I had to chroot, downgrade to r297 and reinstall GRUB.
With this, I noticed the delay too for the first time now. It's not a very long delay, but it appears both on initram and kernel boot.

I don't want to mess with it right now, and r322 doesn't work at all (immediate reboot just before the GRUB menu appears), but I'll see if I can somehow reproduce this.

lenhuppe · 2022-08-26 10:32:05

torvic9 wrote:

Hi,
Since the newest GRUB r322 completely borked the boot, I had to chroot, downgrade to r297 and reinstall GRUB.
With this, I noticed the delay too for the first time now. It's not a very long delay, but it appears both on initram and kernel boot.
I don't want to mess with it right now, and r322 doesn't work at all (immediate reboot just before the GRUB menu appears), but I'll see if I can somehow reproduce this.

I am also seeing the delay when loading the kernel or the ramdisk. It looks to me like Grub is looking for something to load or waiting for something to happen.

It's time for me to learn how Grub actually works. If you look at /etc/default/grub you will see some options. I will start there and see what I can find out.

LanMarc77 · 2022-08-26 16:38:39

I could now also test r322. Also for me it does not change anything in loading speed. Lets wait for the next bisecting step then and be patient.
There is also some discussions here : https://bugs.archlinux.org/task/75701 but at first glance I do not think the two issues are directly connected.

lenhuppe · 2022-08-26 21:18:31

LanMarc77 wrote:

I could now also test r322. Also for me it does not change anything in loading speed. Lets wait for the next bisecting step then and be patient.
There is also some discussions here : https://bugs.archlinux.org/task/75701 but at first glance I do not think the two issues are directly connected.

It's hard to say if they are related w/out knowing more about the root cause. I don't want to to go too far off topic but one thing to consider is the possibility of a systems integration issue.

In my case Grub and OpenZFS are developed under incompatible licenses. I have had to integrate Arch Linux and OpenZFS in my custom installer. Now I am having to integrate
Grub and OpenZFS. Fortunately for me the OpenZFS web site has a lot of information for integrating OpenZFS and Grub which I am currently pouring over.

However the bug reports that I see mention a range of issues. Therefore I am inclined to think that the issue lies with Grub itself.

That said I am confident that the devs and package maintainers are working diligently to resolve whatever the current issue is.

Last edited by lenhuppe (2022-08-26 22:17:35)

Arch Linux

#1 2022-08-19 14:36:38

[SOLVED] Grub update causes delay in loading ramdisk

#2 2022-08-19 14:41:39

Re: [SOLVED] Grub update causes delay in loading ramdisk

#3 2022-08-19 15:07:00

Re: [SOLVED] Grub update causes delay in loading ramdisk

#4 2022-08-19 19:42:19

Re: [SOLVED] Grub update causes delay in loading ramdisk

#5 2022-08-19 21:24:36

Re: [SOLVED] Grub update causes delay in loading ramdisk

#6 2022-08-20 01:19:50

Re: [SOLVED] Grub update causes delay in loading ramdisk

#7 2022-08-21 18:41:21

Re: [SOLVED] Grub update causes delay in loading ramdisk

#8 2022-08-21 20:32:31

Re: [SOLVED] Grub update causes delay in loading ramdisk

#9 2022-08-21 23:33:52

Re: [SOLVED] Grub update causes delay in loading ramdisk

#10 2022-08-22 17:28:27

Re: [SOLVED] Grub update causes delay in loading ramdisk

#11 2022-08-22 17:41:54

Re: [SOLVED] Grub update causes delay in loading ramdisk

#12 2022-08-22 18:17:30

Re: [SOLVED] Grub update causes delay in loading ramdisk

#13 2022-08-22 18:54:49

Re: [SOLVED] Grub update causes delay in loading ramdisk

#14 2022-08-22 19:21:55

Re: [SOLVED] Grub update causes delay in loading ramdisk

#15 2022-08-23 19:56:20

Re: [SOLVED] Grub update causes delay in loading ramdisk

#16 2022-08-24 18:58:24

Re: [SOLVED] Grub update causes delay in loading ramdisk

#17 2022-08-24 20:23:35

Re: [SOLVED] Grub update causes delay in loading ramdisk

#18 2022-08-24 20:36:12

Re: [SOLVED] Grub update causes delay in loading ramdisk

#19 2022-08-24 22:38:15

Re: [SOLVED] Grub update causes delay in loading ramdisk

#20 2022-08-24 23:55:38

Re: [SOLVED] Grub update causes delay in loading ramdisk

#21 2022-08-25 01:40:48

Re: [SOLVED] Grub update causes delay in loading ramdisk

#22 2022-08-26 07:35:16

Re: [SOLVED] Grub update causes delay in loading ramdisk

#23 2022-08-26 10:32:05

Re: [SOLVED] Grub update causes delay in loading ramdisk

#24 2022-08-26 16:38:39

Re: [SOLVED] Grub update causes delay in loading ramdisk

#25 2022-08-26 21:18:31

Re: [SOLVED] Grub update causes delay in loading ramdisk

Board footer