Arch Linux

rudepeople

Member

Registered: 2013-03-10

Posts: 13

Grub 2 install on USB connected nvme causes... strangeness.

I'm trying to install arch on an external USB enclosure with a samsung 980 pro nvme drive in it. Install is very straight forward, however; I've discovered grub is experiencing some kind of quirk I cant explain!

Ultimately I would like to have 2 drives for my laptop. 1 inside the machine I'm running from and one external I can make tweaks to. I want to be able to swap them out from time to time for whatever reason. Cold spare, emergency backup drive, whatever.

I have PDE (Partial Disk Encryption... cant figure out how to encrypt /boot without breaking other things.) with BTRFS on LVM on LUKS with trim and resume support and uefi (though that's kind of implied anymore). The basica install works like a dream, just the grub thing...

Here's my drive layout:

================================================================
# lsblk
	NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
	loop0        7:0    0 706.5M  1 loop /run/archiso/airootfs
	nvme0n1              8:0    0 223.6G  0 disk
	├─nvme0n1p1           8:1    0     1G  0 part  [/boot]
	└─nvme0n1p2           8:2    0 222.6G  0 part
	  └─cryptlvm   254:1    0 222.6G  0 crypt
	    ├─lvm-swap 254:2    0     8G  0 lvm   [SWAP]
		└─lvm-root 254:3    0 214.6G  0 lvm   [/var/log]
		                                      [/home]
											  [/]
	sda              8:0    0 223.6G  0 disk
	├─sda1           8:1    0     1G  0 part  /mnt/boot
	└─sda2           8:2    0 222.6G  0 part
	  └─cryptlvm   254:1    0 222.6G  0 crypt
	    ├─lvm-swap 254:2    0     8G  0 lvm   SWAP
		└─lvm-root 254:3    0 214.6G  0 lvm   /mnt/var/log
		                                      /mnt/home
											  /mnt/
	sdb          8:16   0   1.8T  0 disk
	├─sdb1       8:17   0   1.8T  0 part
	│ └─ventoy 254:0    0 818.3M  0 dm
	└─sdb2       8:18   0    32M  0 part
================================================================

I hope it's obvious, the internal nvme (shown as nvme0n1) and usb nvme (shown as sda) are never decrypted at the same time as this would cause mapper collisions. I typically boot to an ISO in ventoy to play with partitions and I've VERY careful what drive I'm working with.

So the issue is, when I go through my build on the USB drive, the internal drive ceases to be bootable. I can boot back to the ISO and fix the internal drive by running the following:

# cryptsetup luksOpen /dev/[$dev2] cryptlvm
# mount -o subvolid=256 /dev/mapper/lvm-root /mnt
# mount -o subvolid=257 /dev/mapper/lvm-root /mnt/home
# mount -o subvolid=258 /dev/mapper/lvm-root /mnt/var/log
# arch-chroot /mnt /bin/bash
# grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
# nano /etc/default/grub  <-- double check my grub config, usually it's fine.
# mkinitcpio -P linux
# grub-mkconfig -o /boot/grub/grub.cfg
# exit
# umount -a
# reboot

That fixes whatever drive I'm trying to boot to, but ALWAYS breaks the other! The weirdness comes in when I do this to fix the internal drive with the external drive disconnected and vise versa!!! it's like theres some connection between the grub installs that breaks the other disk! I don't get it!

If I take one drive, install it into the laptop, do the build, then move that drive to the USB enclosure and plug in the other drive to the laptop internally and do the repair on that drive (obviously with the USB drive disconnected), BOTH drives work! HOWEVER, if I ever have to invoke grub-mkconfig on the drive in the USB enclosure (even with the internal drive removed), I'm back to square one. what the actual fuck?!?

It seems like there's some boot flag that's getting stripped, but this is uefi... I thought that wasn't a thing anymore!

There used to be a way to specify installing grub to a drive or partition and which drive/partition... I believe that went obsolete with uefi, but it's the only thing that seems to explain this issue. if such a thing still exists, is it possible to still specify the boot partition and not just the install folder?

For a little added clarity, I'm using the laptops boot menu to specify what device to boot to. I don't care if (nor want) either grub instance to see the other.

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,447

Re: Grub 2 install on USB connected nvme causes... strangeness.

From the looks of it you have basically two distinct /boot partitions/ESPs and you basically rotate the config (and the booted GRUB) around of which one is the controlling GRUB as soon as you grub-mkconfig you update one config but not the other... What happens if "for whatever reason I have to rerun grub-mkconfig" you rerun that twice, once for your current /boot and once for the /boot on the other drive? Also in general if you want to be able to willy nilly swap and move drives around and as EFI implementations will remove NVRAM entries of disks that they don't see anymore, what you probably want to do is install grub with the --removable flag to both /boot and instead of booting the GRUB entry you'd boot the named device in the bootmanager of your EFI.

Alternatively, if your EFI is decent and does not remove entries it doesn't see anymore, make distinct --bootloader-id=GRUB (... this is the connection between the installs, your EFI will only store one given GRUB entry with the id "GRUB") for each of the drives, that way you'll have two actually isolated entries, i.e.--bootloader-id=GRUB-Disk1 --bootloader-id=GRUB-Disk2 so this are actually distinctly seperate from each other.

Last edited by V1del (2022-12-09 16:36:22)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,182

Re: Grub 2 install on USB connected nvme causes... strangeness.

You need to use a different bootloader-id for each. Or use --removable on the USB drive.

rudepeople

Member

Registered: 2013-03-10

Posts: 13

Re: Grub 2 install on USB connected nvme causes... strangeness.

Well I'll be a monkey's sysadmin... @V1del and @Scimmia, you were completely correct.

This works:

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=[$uuid-of-boot-partition]

They're fat32 partitions so UUIDs are short. It made picking unique IDs easy and works like a champ! Both drives are happy and bootable.