Arch Linux

newsboost

Member

Registered: 2016-07-24

Posts: 157

[SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Hi all

To make a long story short, I'm trying to setup Arch linux on a new laptop with luks-encrypted boot-partition and lvm-on-luks with detached header... I've succeeded with this maybe 2-3 times before (or done something similar). But now I'm banging my head against the wall because I don't understand why grub-mkconfig uses the wrong UUID... I could of course manually just change the UUID in the /boot/grub/grub.cfg-file - but that isn't really a good solution. The situation is the following:

I think I've done everything as I should - but when I boot I don't even get into the GRUB-menu. Instead it says something like "grub : no such device d9f04646-fff8-4377-..." - I don't even type into the password, to unlock the encrypted boot-partition. I investigated it and found the solution/cause of the problem: And the UUID it's looking for: d9f04646-fff8-4377-... is the decrypted/opened luks-partition, i.e. lsblk -o +UUID shows:

nvme0n1 ...
 - nvme0n1p1 : /boot/efi
 - nvme0n1p2
 - nvme0n1p3
 - nvme0n1p4
 - nvme0n1p5 : 62cd1584-00a5-43...          <=== SHOULD USE THIS INSTEAD (ENCRYPTED UUID)
    --> cryptBoot : d9f04646-fff8-4377...   <=== INCORRECT, THIS IS DECRYPTED
 - nvme0n1p6 : 9ba758e2-9210-...
    --> cryptLVM : P9xAEG-jz96...
       a) volGrp-swap
       b) volGrp-root

So, when I run "mkconfig -o /boot/grub/grub.cfg" GRUB generates some lines, e.g:

search --no-floppy --fs-uuid --set=root d9f04646-fff8-4377...

I think, Instead those lines should say (because the boot-partition isn't unlocked yet, at this stage in the boot-process):

search --no-floppy --fs-uuid --set=root 62cd1584-00a5-43

So far, so good. Now, if we look into my /etc/default/grub file I have something like:

GRUB_CMDLINE_LINUX="loglevel=3 cryptdevice=/dev/nvme0n1p5:cryptBoot cryptkey=rootfs:/root/BACKUP/crypto_keyfile_BOOT.bin cryptdeviceHeader=/dev/nvme0n1p6:cryptLVM:header=/root/BACKUP/cryptLVMheader.img cryptkeyHeader=rootfs:/root/BACKUP/crypto_keyfile_LVM.bin resume=/dev/mapper/volGrp-swap"

My /etc/fstab file contains:

# /dev/mapper/cryptBoot
UUID=d9f04646-fff8-4377...

And I think this is correct, because that's the UUID of the decrypted boot-partition... I've checked and the mkinitcpio.conf FILES=...-section should be ok (checked with lsinitcpio /boot/initramfs-linux.img - this shows me that the key/header files exist - but the problem arises even before this point).

So what I don't understand is (and how to solve the problem?): Why does grub-mkconfig try to use the DECRYPTED UUID (d9f04646-fff8-4377...) instead of the ENCRYPTED UUID (62cd1584-00a5-43..)? It should use the ENCRYPTED UUID and ask me to type in my password - but instead it's looking for a boot partition that doesn't exist and it doesn't allow me to type in the password because it thinks the boot partition doesn't exist... Before arriving here I'm using the "arch-chroot /mnt"-procedure to set things up - but clearly I'm misunderstanding something - please help by explaining to me, what I'm misunderstanding.

I would be grateful for any help that can solve the problem, have been struggling for too many hours now...

Last edited by newsboost (2019-12-31 21:17:52)

mxfm

Member

Registered: 2015-10-23

Posts: 166

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Perhaps grub is located in encrypted partition, so it needs UUID of decrypted device. Or probably the 'luks code' in grub does not recognize partition because of detached header and does not request password. Are you sure you have the same setup in previous times? Probably you had grub on encrypted partition without detached header or grub on unencrypted partition.

Grub developers some time ago decided that everything should be recognized by grub-mkconfig. They decided even that if grub-mkconfig cannot generate config file, such use case is unsupported (this is unfortunate to those who want to use plain dm-crypt - in such case grub-mkconfig cannot know encryption options at all, so it cannot generate config file), they also decided to treat any grub-mkconfig as a bug. In your case this means you can punch them and ask to fix their beloved tool.

newsboost

Member

Registered: 2016-07-24

Posts: 157

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Perhaps grub is located in encrypted partition, so it needs UUID of decrypted device.

That I also don't understand: From where does it figure out which UUID to use for the "search --no-floppy --fs-uuid --set=root (SOME-UUID-HERE)"? I didn't tell it that information specifically I think... Does it look into the /etc/fstab while I'm "arch-chroot /mnt"'ed and then use that UUID? It shouldn't do that. Does it look at the GRUB_CMDLINE_LINUX line and parses the first cryptdevice=...-line? Probably not - because in that line I specifically told it to use /dev/nvme0n1p5 and it should be labelled cryptBoot (/dev/mapper/cryptBoot)? At least that's what I think I've told grub...

Or probably the 'luks code' in grub does not recognize partition because of detached header and does not request password.

No way, I don't believe that. The information about detached header isn't for the boot-partition - it's only for the root-partition (LVM on LUKS) and the boot process fails before even trying to think of anything involving the LVM/root/detached header-partition... Remember GRUB hasn't even come up with it's GRUB-menu yet... And that part is/should be completely familiar to all users of GRUB-boot menu's (at least those with encrypted boot-partitions)...

Are you sure you have the same setup in previous times? Probably you had grub on encrypted partition without detached header or grub on unencrypted partition.

I don't have 100% the same setup, but something very very similar. As before - now I also (at least I think) have grub on encrypted partition without detached header... I don't like having an unencrypted boot-partition (as well as unencrypted root)....

Grub developers some time ago decided that everything should be recognized by grub-mkconfig. They decided even that if grub-mkconfig cannot generate config file, such use case is unsupported (this is unfortunate to those who want to use plain dm-crypt - in such case grub-mkconfig cannot know encryption options at all, so it cannot generate config file), they also decided to treat any grub-mkconfig as a bug. In your case this means you can punch them and ask to fix their beloved tool.

I think I might have made a small mistake somewhere... I didn't write it in my first post, but I also have "GRUB_ENABLE_CRYPTODISK=y". So, the real problem and big prize goes to who knows what information GRUB use to select the UUID of the "cryptBoot"-device instead of the UUID of the nvme0n1p5-partition... Or did I install the bootloader incorrectly? I used something like "grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ArchLinux" - but also here, there ain't no UUID's specically. So GRUB figures out the UUID's based on something I don't understand... Thanks for your input - I hope I've clarified the situation a bit (don't be confused about the detached header-stuff - we aren't even there yet, that stuff takes place AFTER the boot-partition has been unlocked and GRUB tries to search for and unlock a UUID that's incorrect, namely the unlocked boot-partition instead of the locked boot-partition's UUID: /dev/nvme0n1p5)... I must have overlooked something, somewhere...

mxfm

Member

Registered: 2015-10-23

Posts: 166

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

That I also don't understand: From where does it figure out which UUID to use for the "search --no-floppy --fs-uuid --set=root (SOME-UUID-HERE)"? I didn't tell it that information specifically I think... Does it look into the /etc/fstab while I'm "arch-chroot /mnt"'ed and then use that UUID? It shouldn't do that. Does it look at the GRUB_CMDLINE_LINUX line and parses the first cryptdevice=...-line? Probably not - because in that line I specifically told it to use /dev/nvme0n1p5 and it should be labelled cryptBoot (/dev/mapper/cryptBoot)? At least that's what I think I've told grub.

I think it follows information where root partition or probably boot are mounted.

No way, I don't believe that. The information about detached header isn't for the boot-partition - it's only for the root-partition (LVM on LUKS) and the boot process fails before even trying to think of anything involving the LVM/root/detached header-partition... Remember GRUB hasn't even come up with it's GRUB-menu yet... And that part is/should be completely familiar to all users of GRUB-boot menu's (at least those with encrypted boot-partitions)...

Does grub resides on encrypted partition without or with the detached header?

I don't have 100% the same setup, but something very very similar. As before - now I also (at least I think) have grub on encrypted partition without detached header... I don't like having an unencrypted boot-partition (as well as unencrypted root)....

So, grub resided on simple LUKS encrypted partiton (no detached headers)?

I think I might have made a small mistake somewhere... I didn't write it in my first post, but I also have "GRUB_ENABLE_CRYPTODISK=y". So, the real problem and big prize goes to who knows what information GRUB use to select the UUID of the "cryptBoot"-device instead of the UUID of the nvme0n1p5-partition... Or did I install the bootloader incorrectly? I used something like "grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ArchLinux" - but also here, there ain't no UUID's specically. So GRUB figures out the UUID's based on something I don't understand... Thanks for your input - I hope I've clarified the situation a bit (don't be confused about the detached header-stuff - we aren't even there yet, that stuff takes place AFTER the boot-partition has been unlocked and GRUB tries to search for and unlock a UUID that's incorrect, namely the unlocked boot-partition instead of the locked boot-partition's UUID: /dev/nvme0n1p5)... I must have overlooked something, somewhere...

Probably you didn't install grub correctly - that's why UUID points to decrypted partition. Please provide output of relevant commands and post config files.

newsboost

Member

Registered: 2016-07-24

Posts: 157

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

That I also don't understand: From where does it figure out which UUID to use for the "search --no-floppy --fs-uuid --set=root (SOME-UUID-HERE)"? I didn't tell it that information specifically I think... Does it look into the /etc/fstab while I'm "arch-chroot /mnt"'ed and then use that UUID? It shouldn't do that. Does it look at the GRUB_CMDLINE_LINUX line and parses the first cryptdevice=...-line? Probably not - because in that line I specifically told it to use /dev/nvme0n1p5 and it should be labelled cryptBoot (/dev/mapper/cryptBoot)? At least that's what I think I've told grub.

I think it follows information where root partition or probably boot are mounted.

I've been thinking: Maybe I shouldn't have installed GRUB inside the arch-chroot'ed environment? Because in that case the boot partition *IS* unlocked? I don't understand it... If you look at the "GRUB_CMDLINE_LINUX="loglevel=3 cryptdevice=/dev/nvme0n1p5:cryptBoot" I *specifically* told GRUB to use nvme0n1p5 and it's an encrypted device, that when unlocked should be named /dev/mapper/cryptBoot... I don't understand it...

No way, I don't believe that. The information about detached header isn't for the boot-partition - it's only for the root-partition (LVM on LUKS) and the boot process fails before even trying to think of anything involving the LVM/root/detached header-partition... Remember GRUB hasn't even come up with it's GRUB-menu yet... And that part is/should be completely familiar to all users of GRUB-boot menu's (at least those with encrypted boot-partitions)...

Does grub resides on encrypted partition without or with the detached header?

Without detached header - otherwise it would require e.g. inserting a USB-key the whole time, so the encrypted boot-partition should only be unlocked via password (optionally key-file, but that isn't an option at boot-time).

I don't have 100% the same setup, but something very very similar. As before - now I also (at least I think) have grub on encrypted partition without detached header... I don't like having an unencrypted boot-partition (as well as unencrypted root)....

So, grub resided on simple LUKS encrypted partiton (no detached headers)?

Yes, it's as "simple" as that (nvme0n1p5 is a normal luks-encrypted boot-partition)... But I've been thinking: The boot-loader: What is it at the EFI-partition nvme0n1p1 that points to the encrypted boot-partition? How does that work? Maybe it's the bootloader, that's the problem (which I think is installed at the efi-partition, nvme0n1p1)???

I think I might have made a small mistake somewhere... I didn't write it in my first post, but I also have "GRUB_ENABLE_CRYPTODISK=y". So, the real problem and big prize goes to who knows what information GRUB use to select the UUID of the "cryptBoot"-device instead of the UUID of the nvme0n1p5-partition... Or did I install the bootloader incorrectly? I used something like "grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ArchLinux" - but also here, there ain't no UUID's specically. So GRUB figures out the UUID's based on something I don't understand... Thanks for your input - I hope I've clarified the situation a bit (don't be confused about the detached header-stuff - we aren't even there yet, that stuff takes place AFTER the boot-partition has been unlocked and GRUB tries to search for and unlock a UUID that's incorrect, namely the unlocked boot-partition instead of the locked boot-partition's UUID: /dev/nvme0n1p5)... I must have overlooked something, somewhere...

Probably you didn't install grub correctly - that's why UUID points to decrypted partition. Please provide output of relevant commands and post config files.

Sorry, I'm not quite sure which additional config-file(s) or additional output of other relevant commands you wish to see? I can add that /etc/mkinitcpio.conf has a section called "FILES=/..." and these files are needed for handling the encrypted root-volume (with detached header). But after I've been thinking of it, I think it's the installation of the boot-loader that is the problem... I actually almost began to post my config-files, but I knew those wasn't the problem - it would be a waste of our time...

UPDATE: Problem cause and solution:

The problem was that GRUB doesn't understand LUKS2, which is the new standard. It only works with LUKS1, so when I luksFormat'ed my boot partition I did as I always have done - in the future I/we need to supply --type luks1, because luks2 is the new default. You can read more about it e.g. here: https://savannah.gnu.org/bugs/?55093 ; I'm sure this is the problem. I then tried to convert the luks2-boot partition several times using different methods ("cryptsetup convert --type luks1 ...") but it didn't work (google "downgrade luks2 to luks1"), the problem was apparantly that my keyslot 0 was "incompatible" as it was of type luks2. There's a thread here were they experience the same: https://unix.stackexchange.com/question … -version-1 - but I also tried " cryptsetup luksConvertKey --pbkdf=pbkdf2 luks2.img" which seemd to work for them - but not for me. It is actually also written in the arch wiki at https://wiki.archlinux.org/index.php/Dm … ire_system - quote: "Warning: GRUB does not support LUKS2. Use LUKS1 (--type luks1) on partitions that GRUB needs to access" - but that wasn't as big a problem before as LUKS1 was default...

Now I think other people will face the same challenges as I did in the future (I've spend >20 hours on this!), hence this explanation so others can benefit of my experiences. Maybe I should have tried harder to downgrade, but I ended up luksFormatting nvme0n1p5 with --type luks1. This caused some problems with "grub-mkconfig -o /boot/grub/grub.cfg" which didn't find the linux-partition, only the windows. After a while trying various stuff I realized I was missing /boot/vmlinuz- (obviously a consequence of wiping/reformatting)... I tried "pacman -S linux" from inside the arch-chroot'ed environment but that also failed - luckily (as it should) with CLEAR error messages/warnings. Then I exited the chrooted environment, and did pacstrap /mnt base linux - then arch-chroot again - redid the "pacman -S linux" - and now the /boot/vmlinuz-files where in place (the output messages also indicated success at visually I could see the files were inside /boot/vmlinuz-...). Then grub-install - something like that - reboot - and FINALLY: Now I've arrived at the "Welcome to GRUB!"-prompt. From this point on, I hope/expect to add the detached header-stuff and continue more or less as I normally have done.

Thanks for your help! I also hope these new findings + explanation can be of help to other people in the future, at least until GRUB begins to warn about the fact that it doesn't support LUKS2, *SHAME ON YOU, NASTY GRUB!* :-)

mxfm

Member

Registered: 2015-10-23

Posts: 166

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Information about luks1 in grub and about changes in base package (removing linux package) was mentioned in news and in the wiki.

newsboost

Member

Registered: 2016-07-24

Posts: 157

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Information about luks1 in grub and about changes in base package (removing linux package) was mentioned in news and in the wiki.

I'm not sure what you mean by this? The problem as I see it, isn't really Arch Linux - but Grub should come up with a warning/error, when someone (like me - because cryptsetup now defaults to luks2) tries to use a luks2-encrypted boot partition, when that isn't supported - instead of silently just using another UUID... I think this is bad program behaviour, because you normally would expect the program to do what you tell it unless it can't. If it can't I expect the program to output or use error/warning messages to tell the reason for this (and maybe how to solve the problem, if it can be done short)... Many people like me don't install grub on encrypted partitions very often so I/we don't read Grub developer news and normally I don't expect that to be necessary, for any decent linux program. I didn't know linux was in the base package before and has been moved out. I don't see this as a real problem... But thanks again for helping (I was really frustrated in the end, couln't understand a word of what was wrong, so it was good someone tried to help)...

mxfm

Member

Registered: 2015-10-23

Posts: 166

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Information about luks1 in grub and about changes in base package (removing linux package) was mentioned in news and in the wiki.

I'm not sure what you mean by this? The problem as I see it, isn't really Arch Linux - but Grub should come up with a warning/error, when someone (like me - because cryptsetup now defaults to luks2) tries to use a luks2-encrypted boot partition, when that isn't supported - instead of silently just using another UUID...

I mean that information about GRUB not supporting luks2 was included in the wiki. Meanwhile there is issue in grub bugzilla and patches were sent to the mailing list.

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [SOLVED] grub-mkconfig used the wrong UUID (LUKS-encrypt. boot-part.)

Grub developers some time ago decided that everything should be recognized by grub-mkconfig. They decided even that if grub-mkconfig cannot generate config file, such use case is unsupported (this is unfortunate to those who want to use plain dm-crypt - in such case grub-mkconfig cannot know encryption options at all, so it cannot generate config file), they also decided to treat any grub-mkconfig as a bug. In your case this means you can punch them and ask to fix their beloved tool.

I defy this claim.

It is easier, simpler, and more reliable to write your own, and it's a perfectly reasonable use case, because at the end of the day, grub-mkconfig merely does its best guess at writing a config file. The problem with writing your own is not that the configuration primitives you need don't exist, it's that the documentation is terrible. How can the grub developers possibly refuse to support a grub command not working as expected, if grub-mkconfig relies on this...

Moreover, grub-mkconfig "officially" supports /etc/grub.d/40_custom (to add arbitrary content to) and /etc/grub.d/41_custom (to automatically source /boot/grub/custom.cfg).

I've written up some guidance on doing it properly. See https://wiki.archlinux.org/index.php/Us … figuration

Last edited by eschwartz (2020-01-01 15:00:16)

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)