You are not logged in.

#1 2020-08-04 12:17:45

coinpouch
Member
Registered: 2017-03-20
Posts: 10

LUKS on LVM, UEFI, unable to find device after kernel upgrade

Hi,

After a recent kernel upgrade + boot, I've been unable to boot, where it doesn't get up to the point of asking me for my crypt password. I have the Arch News package installed, and nothing popped up during this particular upgrade.

Everything as normal from a live USB.

Error message when booting, after selecting Arch:

:: running early hook [udev]
Starting version 245.7-1-arch
:: running early hook [lvm2]
:: running early hook [udev]
:: Triggering uevents...
:: running hook [keymap]
:: Loading keymap...done.
:: running hook [encrypt]
Waiting 10 seconds for device /dev/disk/by-uuid/<uuid of /dev/mapper/MyVolGroup-cryptroot here>...
[ 4.203606] sd 0:0:0:0: [sda] No caching mode page found # OP comment - sda is my live usb
[ 4,203631] sd 0:0:0:0: [sda] Assuming drive cache: write through
:: running hook [encrypt]
Waiting 10 seconds for device /dev/disk/by-uuid/<uuid of /dev/mapper/MyVolGroup-cryptroot here>...
Waiting 10 seconds for device /dev/mapper/root ...
Error device '/dev/mapper/root' not found. Skipping fsck.
:: mounting '/dev/mapper/root' on real root
mount: /new_root: special device /dev/mapper/root does not exist.
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]#

At this point, it doesn't seem like any of my logical volumes show up

ls /dev/
ls /dev | grep My 
# nothing

ls /dev/mapper
# control

ls /dev/ | grep nvme
# nvme0

Does the fact nvme shows up but the LVM volumes themselves don't mean something is off with the lvm2 dependencies/hooks/etc.?

Relevant bits of /etc/default/grub:

# I know on the current version of the arch wiki, it says lvm is unneeded as grub-mkconfig auto detects if lvm is needed - included it as I was out of ideas for what was wrong
GRUB_PRELOAD_MODULES="part_gpt part_msdos lvm"
GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=<uuid of /dev/mapper/MyVolGroup-cryptroot>:root /dev/mapper/root"
GRUB_CMDLINE_LINUX = ""
GRUB_ENABLE_CRYPTODISK=y

Relevant bits of /etc/fstab:

/dev/mapper/root / ext4 defaults 0 1
/dev/nvme0n1p2 /boot ext4 defaults 0 2
/dev/mapper/tmp /tmp tmpfs defaults 0 0
/dev/mapper/swap none swap sw 0 0

Relevant bits of /etc/crypttab:

swap /dev/MyVolGroup/cryptswap /dev/urandom swap,cipher=aes-xts-plain64,size=256
tmp /dev/MyVolGroup/crypttmp /dev/urandom tmp,cipher=aes-xts-plain64,size=256 
# tmp /dev/MyVolGroup/crypttmp /dev/unradom tmp,cipher=aes-xts-plain64,size=256 # this line had a typo, these config snippets were typed out by hand :<

Relevant bits of /etc/mkinitcpio.conf

MODULES=()
BINARIES=()
FILES=()
HOOKS=(base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 encrypt filesystems fsck)

I've arch-chroot'ed from a live USB and rerun mkinitcpio -P, and reinstalled grub/regenerated grub config after checking the above many times without any luck.
The following steps complete without a hitch:

cryptsetup open /dev/mapper/MyVolGroup-cryptroot root
mount /dev/mapper/root /mnt
mount /dev/nvme0n1p2 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
arch-chroot /mnt
###
mkinitcpio -P
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootload-id=GRUB --recheck
grub-mkconfig -o /boot/grub/grub.cfg
exit
umount -R /mnt
reboot

I've run into this same problem many times in the past, but from memory, either a rebuild of things from the above line solved the issue, or lvm2 had magically disappeared from my mkinitcpio.conf HOOKS and adding that back in/rebuilding the initramfs images resolved the problem, but those steps haven't gotten me anywhere this time.

At my wits on end on what could be wrong - any tips on how to continue troubleshooting this would be greatly appreciated! Please let me know what important info I may be missing from the post to diagnose the issue further.

Last edited by coinpouch (2020-08-04 12:46:51)

Offline

#2 2020-08-04 12:44:14

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: LUKS on LVM, UEFI, unable to find device after kernel upgrade

your ls /dev | grep nvme did not list partitions, unless you left them out of your post, that's very odd

things to check in the initramfs rescue shell

cat /proc/cmdline   # all kernel parameters as expected?
cat /proc/partitions # all disks and partitions found as expected?
ls -l /dev/disk/by-uuid # if udev is running and devices present, you should find your UUIDs here

if there are no unexpected results until here, you can try enable things manually

lvm vgscan --mknodes # optional, creates /dev/mapper/control if udev did not
lvm lvchange -a ly VG/root
lvm vgscan --mknodes # optional, creates /dev/mapper/VG-LV and /dev/VG/LV if udev did not
cryptsetup open /dev/VG/LV cryptlv
...

there is a typo in your crypttab (or typo in your post) unrandom -> urandom

If you have LUKS on top of LVM (instead of LVM on top of LUKS), the lvm2 hook should be before encrypt hook. It should not make a difference currently since lvm2 hook uses lvmetad, but otherwise the order of hooks is important.

If you have `use_lvmetad=0` in `lvm config` (check within arch-chroot), that may be the reason why the lvm2 hook does not work for you.

Last edited by frostschutz (2020-08-04 12:46:37)

Offline

#3 2020-08-04 13:02:00

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,429

Re: LUKS on LVM, UEFI, unable to find device after kernel upgrade

From which kernel did you update? Who is your nvme drive vendor? Sounds related to: https://bbs.archlinux.org/viewtopic.php?id=257375

Online

#4 2020-08-04 13:04:37

coinpouch
Member
Registered: 2017-03-20
Posts: 10

Re: LUKS on LVM, UEFI, unable to find device after kernel upgrade

Thanks for the pointers @frostchutz! Went back and double checked the issues/typos you pointed out to make sure the original doesn't have them and reran mkinitpcio -P, and reinstalled/created grub and grub config.

your ls /dev | grep nvme did not list partitions, unless you left them out of your post, that's very odd

Yeah, the ls /dev | grep nvme only listed a single result - nvme0


things to check in the initramfs rescue shell
cat /proc/cmdline   # all kernel parameters as expected?

I think this looks correct (may be mistaken though?)

BOOT_IMAGE=/vmlinuz-linux-lts root=UUID=<uuid of the decrypted LUKS partition at /dev/mapper/MyVolGroup-cryptroot> rw cryptdevice=UUID=<uuid of /dev/mapper/MyVolGroup-cryptroot>:root root=/dev/mapper/root

cat /proc/partitions # all disks and partitions found as expected?

Looks like this is where it falls down?

major minor #blocks name
8 0 7828480 sda
8 1 7827439 sda1

All my machine's nvme0n.... drives are missing here


If you have LUKS on top of LVM (instead of LVM on top of LUKS), the lvm2 hook should be before encrypt hook. It should not make a difference currently since lvm2 hook uses lvmetad, but otherwise the order of hooks is important.

Yep definitely - just realised I accidentally included an extra 'encrypt' hook in my typed out version - both before and after lvm2, which isn't in the actual config after checking against the config on my machine

ls -l /dev/disk/by-uuid # if udev is running and devices present, you should find your UUIDs here

Only the sda device is present

lrwxrwxrwx 1 0 0 10 Aug 4 22:56 7654-40DA -> ../../sda1

lvm commands

I unfortunately wasn't able to proceed with these as none of the physical or virtual devices required for this were actually visible in the emergency shell

If you have `use_lvmetad=0` in `lvm config` (check within arch-chroot), that may be the reason why the lvm2 hook does not work for you.

`lvm config` is showing `use_lvmetad=1`

Offline

#5 2020-08-04 13:19:27

coinpouch
Member
Registered: 2017-03-20
Posts: 10

Re: LUKS on LVM, UEFI, unable to find device after kernel upgrade

V1del wrote:

From which kernel did you update?

Judging from `/var/cache/pkg`, it looks like I upgraded from linux-5.7.12 (the latest cached version I have - though I also have 5.7.11 in the cache)
And for lts, linux-lts-5.4.54

Who is your nvme drive vendor?

This is the make/model of the NVMe drive:
Silicon Power 1TB NVMe PCIe P34A60 Gen3x4 M.2 2280 TLC SSD (SP001TBP34A60M28)

That definitely sounds related, thanks - will look into patching those things/doing a downgrade.

Last edited by coinpouch (2020-08-04 13:24:01)

Offline

#6 2020-08-04 13:30:29

coinpouch
Member
Registered: 2017-03-20
Posts: 10

Re: LUKS on LVM, UEFI, unable to find device after kernel upgrade

To confirm - rolled back to 5.7.7 as suggested in https://bbs.archlinux.org/viewtopic.php?id=257375, that solved the problem

Offline

Board footer

Powered by FluxBB