Accidentally installed GRUB to bootable USB - cant boot without

jordaniza · 2022-09-01 10:04:52

Hi.

So after the recent issues with Grub, I was unable to boot into my system (5.19.5.arch1-1), grub (2:2.06.r322.gd9b4638c5-3) was not appearing.

I booted from a live USB to try and remedy my installation. Below is the result of `fdisk -l`, dev/nvmeXXX is my SSD while the USB device is at /dev/sda:

Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 980 PRO 1TB                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 524CFFC2-B220-9546-BDCC-50129630048C

Device           Start        End    Sectors  Size Type
/dev/nvme0n1p1    2048    1050623    1048576  512M EFI System
/dev/nvme0n1p2 1050624 1953525134 1952474511  931G Linux LVM


Disk /dev/mapper/volgroup0-lv_root: 439.45 GiB, 471859200000 bytes, 921600000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/volgroup0-lv_home: 488.28 GiB, 524288000000 bytes, 1024000000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 28.64 GiB, 30752636928 bytes, 60063744 sectors
Disk model: Ultra           
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x22372fe3

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *         64 1587199 1587136  775M  0 Empty
/dev/sda2       1587200 1763327  176128   86M ef EFI (FAT-12/16/32)

I needed to provide an EFI directory to grub-install. I read the wiki and also followed suggestions in this thread and decided to mount the filesystems as follows:

# lvm needs mounting of logicial volume
mount /dev/volgroup0/lv_root /mnt
mkdir /mnt/boot/efi
mount /dev/nvme0n1p1 /mnt/boot/efi

I then chrooted into the /mnt directory and ran:

grub-install --efi-directory=/boot/efi --removable

This succeeded but then when I opened my system, I was dumped into the `grub rescue` menu

# might not be exact, doing this from memory
symbol 'grub_malloc' not found

Interestingly, this grub issue prevents me from accessing the BIOS settings to boot into a USB unless I very quickly hit `esc`.

Also, super weirdly, when I boot into the BIOS menu and select one-time-boot into the removable USB, grub works as it used to...I can select my Arch distro and then the full system loads correctly (I am using it now), this includes the DE and GUI. However this means I cannot boot into the 'fresh' Arch ISO anymore (where I would have the black screen and have to mount the drives then chroot in) selecting a distro just goes straight to the KDE login screen.

So I'm in this a place where I've clearly borked my system somehow by screwing up grub, but installing it instead on the USB.

Logging in, I tried re-reading some additional help topics, and eventually tried to rebuild grub again, this time from within my working system:

su
mount /dev/nvme0n1p2 /boot/efi
grub-install  --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB --no-nvram --removable
grub-mkconfig -o /boot/grub/grub.cfg

This again built without issues but I am in the same place as before. Atm I am a bit stuck and would appreciate help on next steps.

I noticed a few other things that might be helpful:

1. /boot contains both /EFI, and an /efi folder
2. My efi vars are at

 /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)

V1del · 2022-09-01 10:22:12

If you are booting an entry that is literally called GRUB, then by enforcing --removable you are never updating that entry.

FWIW since you had to explicitly create a directory /boot/efi chances are that your working setup used to have the esp on /boot directly, thus alleviating GRUB of the need to know about your LVM setup. You can just set up the ESP on /boot directly again. By not doing so you technically need GRUB to know about LVM which might need some more configuration or potentially run into more bugs.

If you had a working setup, and then need to create directories from your root chroot then you are changing something from the working setup. The easiest way to know how your setup used to work if you don't remember is looking at your /etc/fstab. If you didn't change that you need to mount things as they are listed there ( ... mount -a is your friend ) and then relevantly adjust the corresponding commands

Last edited by V1del (2022-09-01 10:24:23)

jordaniza · 2022-09-01 12:17:26

Ok cool, so...

etc/fstab gives:

# /dev/mapper/volgroup0-lv_root
UUID=67376f00-d73c-4e18-b1d9-d24919cd2a5b   /            ext4   rw,relatime   0 1 

# /dev/mapper/volgroup0-lv_home
UUID=3040f2e5-5aaa-43cc-a9c8-8421ce401327   /home        ext4   rw,relatime   0 2 

/swapfile                                   none         swap   sw            0 0

So based on your suggestion, I should run:

# mount the filesystem as specified in /etc/fstab
mount -a 

# install a new version of grub
grub-install  --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

# regen the grub config file
grub-mkconfig -o /boot/grub/grub.cfg

I don't see where else to place the ESP in --efi-directory. From what I can see...should be in either /boot/efi or /boot/EFI?

Last edited by jordaniza (2022-09-01 12:18:00)

V1del · 2022-09-01 12:25:08

That depends on how you've set it up... With the ESP mounted, let's take /boot/efi FWIW, what is your output of

tree /boot #Might have to install tree
efibootmgr -uv

Logically from your attempted progression it should be mounted on /boot and all of GRUB and it's config and your kernel images are there.

Last edited by V1del (2022-09-01 12:27:37)

seth · 2022-09-01 12:38:58

He might have "updated" a kernel and now installed grub into that location already.
Let's see where you're booting from:

cat /proc/cmdline

jordaniza · 2022-09-01 12:44:19

Struggling a little with understanding if the ESP is mounted here - I've executed the commands but one core issue I have is that I'm not completely sure if the mount was done correctly.

I ran the commands twice, first I ran without any mounting, then I did what I think is the correct mount:

mount /dev/nvme0n1p1 /mnt/boot/efi

Output of tree /boot after mounting

Output of tree /boot

Diff is then:

|-- efi
|   `-- EFI
|       |-- BOOT
|       |   `-- BOOTX64.EFI
|       |-- GRUB
|       |   `-- grubx64.efi
|       `-- grub_uefi
|           `-- grubx64.efi

My guess then, is that:

- I should be mounting at /boot, as opposed to /boot/efi
- the ESP is /boot/EFI, meaning the command should be:

grub-install  --target=x86_64-efi --efi-directory=/boot/EFI --bootloader-id=GRUB

Output of efibootmgr -uv:

BootCurrent: 0004
Timeout: 2 seconds
BootOrder: 0002,0003,0001,0000,0004
Boot0000* Linux Boot Manager    HD(1,GPT,bfe75c45-a5b6-4b91-9719-2e7ea2def6a0,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 45 5c e7 bf b6 a5 91 4b 97 19 2e 7e a2 de f6 a0 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0001* UiApp MemoryMapped(11,0x840000,0xffffff)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
      dp: 01 03 18 00 0b 00 00 00 00 00 84 00 00 00 00 00 ff ff ff 00 00 00 00 00 / 04 06 14 00 21 aa 2c 46 14 76 03 45 83 6e 8a b6 f4 66 23 31 / 7f ff 04 00
Boot0002* grub_uefi     HD(1,GPT,6da32840-ec1a-9b4b-b4d3-fb6cd5e35a7b,0x800,0x100000)/File(\EFI\grub_uefi\grubx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 40 28 a3 6d 1a ec 4b 9b b4 d3 fb 6c d5 e3 5a 7b 02 02 / 04 04 3a 00 5c 00 45 00 46 00 49 00 5c 00 67 00 72 00 75 00 62 00 5f 00 75 00 65 00 66 00 69 00 5c 00 67 00 72 00 75 00 62 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0003* Pop!_OS 21.10 HD(1,GPT,bfe75c45-a5b6-4b91-9719-2e7ea2def6a0,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 45 5c e7 bf b6 a5 91 4b 97 19 2e 7e a2 de f6 a0 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0004* SanDisk Ultra         PciRoot(0x0)/Pci(0x14,0x0)/USB(19,0)걎脈鼑䵙຅᫢ⱒ뉙      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 14 / 03 05 06 00 13 00 / 7f ff 04 00
    data: 4e ac 08 81 11 9f 59 4d 85 0e e2 1a 52 2c 59 b2

And after mounting:

BootCurrent: 0004
Timeout: 2 seconds
BootOrder: 0002,0003,0001,0000,0004
Boot0000* Linux Boot Manager    HD(1,GPT,bfe75c45-a5b6-4b91-9719-2e7ea2def6a0,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 45 5c e7 bf b6 a5 91 4b 97 19 2e 7e a2 de f6 a0 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0001* UiApp MemoryMapped(11,0x840000,0xffffff)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
      dp: 01 03 18 00 0b 00 00 00 00 00 84 00 00 00 00 00 ff ff ff 00 00 00 00 00 / 04 06 14 00 21 aa 2c 46 14 76 03 45 83 6e 8a b6 f4 66 23 31 / 7f ff 04 00
Boot0002* grub_uefi     HD(1,GPT,6da32840-ec1a-9b4b-b4d3-fb6cd5e35a7b,0x800,0x100000)/File(\EFI\grub_uefi\grubx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 40 28 a3 6d 1a ec 4b 9b b4 d3 fb 6c d5 e3 5a 7b 02 02 / 04 04 3a 00 5c 00 45 00 46 00 49 00 5c 00 67 00 72 00 75 00 62 00 5f 00 75 00 65 00 66 00 69 00 5c 00 67 00 72 00 75 00 62 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0003* Pop!_OS 21.10 HD(1,GPT,bfe75c45-a5b6-4b91-9719-2e7ea2def6a0,0x800,0x100000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 45 5c e7 bf b6 a5 91 4b 97 19 2e 7e a2 de f6 a0 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0004* SanDisk Ultra         PciRoot(0x0)/Pci(0x14,0x0)/USB(19,0)걎脈鼑䵙຅᫢ⱒ뉙      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 14 / 03 05 06 00 13 00 / 7f ff 04 00
    data: 4e ac 08 81 11 9f 59 4d 85 0e e2 1a 52 2c 59 b2

jordaniza · 2022-09-01 12:56:36

seth wrote:

He might have "updated" a kernel and now installed grub into that location already.
Let's see where you're booting from:
cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/volgroup0-lv_root rw loglevel=3 quiet ibt=off ec_sys.write_support=1

seth · 2022-09-01 13:55:21

You're currently booting from the root partition and there're no kernels installed on /dev/nvme0n1p1 so if you want to use that as ESP (did you ever remove kernel images and initramfs from what's currently /boot/efi?) you'll have to re-install the kernel (for that location) as well.
Do you currently have https://wiki.archlinux.org/title/GRUB#LVM ?

jordaniza · 2022-09-01 15:05:51

preload is set to:

GRUB_PRELOAD_MODULES="part_gpt part_msdos"

So I guess it should be:

GRUB_PRELOAD_MODULES="part_gpt part_msdos lvm"

What I don't understand, is that I've had no issues with this for the past 8 months. Maybe I deleted or moved something?

The intramfs and linuz files are in the /boot dir though:

/boot
|-- .....
|-- initramfs-linux-fallback.img
|-- initramfs-linux-lts-fallback.img
|-- initramfs-linux-lts.img
|-- initramfs-linux.img
|-- intel-ucode.img
|-- vmlinuz-linux
`-- vmlinuz-linux-lts

jordaniza · 2022-09-01 15:21:52

Also, I'm not sure if this helps, but I mounted the USB on `/run/media` and ran:

tree /run/media

/run/media
|-- EFI
|   |-- BOOT
|   |   |-- BOOTIA32.EFI
|   |   |-- BOOTx64.EFI
|   |   `-- grub.cfg
|   `-- arch
|       `-- grubx64.efi
|-- arch
|   `-- boot
|       |-- amd-ucode.img
|       |-- intel-ucode.img
|       `-- x86_64
|           |-- initramfs-linux.img
|           `-- vmlinuz-linux
|-- grub
|-- loader
|   |-- entries
|   |   |-- 01-archiso-x86_64-linux.conf
|   |   |-- 02-archiso-x86_64-speech-linux.conf
|   |   |-- 03-archiso-x86_64-ram-linux.conf
|   |   `-- 04-archiso-x86_64-ram-speech-linux.conf
|   `-- loader.conf
|-- shellia32.efi
`-- shellx64.efi

So I think there are some kernel images on the USB, are these meant to be in my root or boot partitions?

seth · 2022-09-01 15:35:56

w/ /boot/EFI present and no lvm support in grub (at least not through the preload modules) one might infer that you previously had the "EFI System" partition mounted at /boot and must have deleted the kernel image from there at some point.
Another explanation could be that you never used grub to boot the system but the efi bootloader.

No, idea. But you'll have to make up your mind on what you'd like to do and then pursuit that.
And inb4:

V1del wrote:

You can just set up the ESP on /boot directly again. By not doing so you technically need GRUB to know about LVM which might need some more configuration or potentially run into more bugs.

jordaniza · 2022-09-01 16:23:06

Ok great and I checked my old setup (dug up some old videos and articles I remember leaning on).

I previously mounted the /dev/nvme0n1p1 EFI Filesystem on /boot/EFI
then I installed grub on the /boot/EFI directory.

I'd like to keep that setup as I've had no problems up until now with it.

In that case all that is left is your comment on deleting the kernel image. If I look inside my /boot folder, I can see the images in there:

/boot
|-- initramfs-linux-fallback.img
|-- initramfs-linux-lts-fallback.img
|-- initramfs-linux-lts.img
|-- initramfs-linux.img
|-- intel-ucode.img
|-- vmlinuz-linux
|-- vmlinuz-linux-lts
`-- /EFI
     ` -- (empty)

So I'm thinking:

umount /boot/efi and delete the empty directory

mount EFI directory at /boot/EFI

run the new grub-install and grub.cfg commands - will the linux images in /boot be available to /boot/EFI and grub?

Last edited by jordaniza (2022-09-01 16:24:18)

seth · 2022-09-01 20:00:05

At least w/ the lvm module preloaded, they should™
W/o I frankly don't know - you could just try and figure

jordaniza · 2022-09-02 05:28:25

Thanks for your help. I have enough to go on now and am much, much clearer on where I think I messed up.

I will be installing arch iso to a second USB and keeping the current one as is, so I can do a recovery chroot if I screw something up.

Will report back here when I'm done.

Arch Linux

#1 2022-09-01 10:04:52

Accidentally installed GRUB to bootable USB - cant boot without

#2 2022-09-01 10:22:12

Re: Accidentally installed GRUB to bootable USB - cant boot without

#3 2022-09-01 12:17:26

Re: Accidentally installed GRUB to bootable USB - cant boot without

#4 2022-09-01 12:25:08

Re: Accidentally installed GRUB to bootable USB - cant boot without

#5 2022-09-01 12:38:58

Re: Accidentally installed GRUB to bootable USB - cant boot without

#6 2022-09-01 12:44:19

Re: Accidentally installed GRUB to bootable USB - cant boot without

#7 2022-09-01 12:56:36

Re: Accidentally installed GRUB to bootable USB - cant boot without

#8 2022-09-01 13:55:21

Re: Accidentally installed GRUB to bootable USB - cant boot without

#9 2022-09-01 15:05:51

Re: Accidentally installed GRUB to bootable USB - cant boot without

#10 2022-09-01 15:21:52

Re: Accidentally installed GRUB to bootable USB - cant boot without

#11 2022-09-01 15:35:56

Re: Accidentally installed GRUB to bootable USB - cant boot without

#12 2022-09-01 16:23:06

Re: Accidentally installed GRUB to bootable USB - cant boot without

#13 2022-09-01 20:00:05

Re: Accidentally installed GRUB to bootable USB - cant boot without

#14 2022-09-02 05:28:25

Re: Accidentally installed GRUB to bootable USB - cant boot without

Board footer