Arch Linux

GuloGuloDesu

Member

Registered: 2013-07-18

Posts: 35

EFI, LUKS, and LVM grub-install error: unknown filesystem

Proxmox VE 8.1.3 (qemu)
Arch ISO from 01/01/2024

I am trying to do a base install of Arch on a VM. I am able to install Arch just fine without using any encryption. But when I go to use LUKS to encrypt everything, whenever I run grub-install, I receive an unknown filesystem error. I have read through a bunch of forum posts about this, and so far none of them have helped.
I manually ran all of these commands on my desktop and on my laptop to do the install, and they are both functioning, so I think it has something to do with something missing on the VM.
I am assuming that the error: unknown filesystem that I receive when I run grub-install is what is preventing the system from booting, as it isn't an error that I receive on my desktop or my laptop. Nor is it an error that I receive when I am not using encryption.

Here is the first script that I am running:

######### Global variable definitions #########
FULL_SCRIPT_PATH="$(realpath "$0")"
SCRIPT_NAME="$(basename "$0")"
DIR_PATH="$(dirname $FULL_SCRIPT_PATH)"


######### Function definitions #########


######### Main Script #########
logger -s -t $SCRIPT_NAME -p local3.info "Script started"

#read -p "Press any key to continue..." -n1 -s

# List the drives that are at least 1 GB in size
SYSTEM_DRIVE="$(fdisk -l | grep 'GiB' | awk '{print $2}' | sed 's/.$//')"
logger -s -t $SCRIPT_NAME -p local3.info "Found the drive ${SYSTEM_DRIVE} to use for setup"

# Delete all partitions to clear
sgdisk --zap-all "${SYSTEM_DRIVE}"

# Create a new EFI boot partition
sgdisk --new=1::+512M --typecode 1:ef00 "${SYSTEM_DRIVE}"

# Create a new Linux partition that is the rest of the drive space
sgdisk --new=2::0 "${SYSTEM_DRIVE}"

logger -s -t $SCRIPT_NAME -p local3.info "Drive has been successfully partitioned"

# Enable cryptsetup
modprobe dm-crypt

# Encrypt the drive
cryptsetup -h SHA512 -i 11000 -c aes-xts-plain -y -s 512 luksFormat --type luks1 "${SYSTEM_DRIVE}2"

# Passwords will be typed by Packer

# Unlock the enctypted drive
cryptsetup open "${SYSTEM_DRIVE}2" lvmvirt

# Password will be typed by Packer

# Create a new physical volume
#pvcreate "${SYSTEM_DRIVE}2"
pvcreate /dev/mapper/lvmvirt

# Create a volume group on the new physical volume
#vgcreate vglinux "${SYSTEM_DRIVE}2"
vgcreate vglinux /dev/mapper/lvmvirt

# Create the logical volume on the new volume group that is on the new physical volume
lvcreate -l 100%FREE -n lvlinux vglinux

# Format the new logical volume on the new volume group that is on the new physical volume
mkfs.ext4 /dev/mapper/vglinux-lvlinux

# Format the first partition drive
mkfs.fat -F32 "${SYSTEM_DRIVE}1"

logger -s -t $SCRIPT_NAME -p local3.info "All physical and logical partitions have been created and formatted"

# Mount the primary partition
mount /dev/mapper/vglinux-lvlinux /mnt

# Make the boot and EFI directories
mkdir -p /mnt/boot/efi

# Mount the physical partition to the EFI directory
mount -t vfat "${SYSTEM_DRIVE}1" /mnt/boot/efi

logger -s -t $SCRIPT_NAME -p local3.info "Directories and mount points created"

# Install Arch packages (linux-firmware should be used on physical systems)
pacstrap -K /mnt base base-devel linux

logger -s -t $SCRIPT_NAME -p local3.info "Installation of Arch apps completed"

# Generate the FStab for the UUID's
genfstab -U /mnt >> /mnt/etc/fstab

logger -s -t $SCRIPT_NAME -p local3.info "fstab created"

cp arch_chroot.sh /mnt/

logger -s -t $SCRIPT_NAME -p local3.info "Starting chroot"

# Chroot into the Arch installation
#arch-chroot /mnt /bin/bash /arch_chroot.sh
arch-chroot /mnt

logger -s -t $SCRIPT_NAME -p local3.info "Chroot has completed"

# Unmount the mount points
#umount -R /mnt

# Reboot the system after installation
#reboot

######### Success Exit code for the script #########
logger -s -t $SCRIPT_NAME -p local3.info "Script completed"
#exit 0

I am having to do something a little different, because for whatever reason, when I add a command after arch-chroot, it will not enter chroot. Even if it is just arch-chroot /mnt ls it will not go into chroot, and just execute ls. Weird problem that I will probably create another post for once I get the rest of the installation working. This script was working for the past several weeks with the command after arch-chroot (I have spun up around a dozen VM's using it so far).

Once I am in chroot, I execute this script:

######### Global variable definitions #########
FULL_SCRIPT_PATH="$(realpath "$0")"
SCRIPT_NAME="$(basename "$0")"
DIR_PATH="$(dirname $FULL_SCRIPT_PATH)"


######### Function definitions #########


######### Main Script #########

#read -p "Press any key to continue..." -n1 -s
echo "arch_chroot scripts has started"

# Add a new users
useradd -m -G wheel user_1
useradd -m -s /sbin/nologin aur_installer

# Set the root password
passwd

# Set the password for the new users
passwd user_1

echo "Users create with passwords"

# Set the Time Zone
ln -sf /usr/share/zoneinfo/US/Pacific /etc/localtime

# Sync the hardware clock to the system clock
hwclock --systohc

# Set localization to en_US.UTF-8
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf

echo "Timezones, locales and time has been updated"

# Set the hostname
echo arch01 > /etc/hostname

# Deteremine the processor type to determine the ucode that needs to be installed
PROCESSOR_VENDOR_ID=$(lscpu | grep Vendor | grep -V BIOS)

if [[ $PROCESSOR_VENDOR_ID =~ "AMD" ]]
then
    UCODE="amd-ucode"
else
    UCODE="intel-ucode"
fi

# Update pacman
pacman -Sy

# Install the EFI boot manager
pacman --noconfirm -S efibootmgr grub lvm2 vim dhclient networkmanager nm-connection-editor openssh git python3 linux-firmware ${UCODE}
# Or amd-ucode Should probably build a check for which one to install

# Create keyfile to avoid second decryption password request
dd bs=512 count=4 if=/dev/random of=/root/cryptlvm.keyfile iflag=fullblock

# Change permissions on the new keyfile
chmod 000 /root/cryptlvm.keyfile

echo "Cryptlvm keyfile has been created"

# Populate vconsole.conf file for font into during bootup
echo "FONT=lat2-16" >> /etc/vconsole.conf
echo "FONT_MAP=8859-2" >> /etc/vconsole.conf

# Add lvm2 to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/fsck/lvm2 &/g' /etc/mkinitcpio.conf

# Add resume to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/lvm2/resume &/g' /etc/mkinitcpio.conf

# Add encrypt to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/filesystems/encrypt &/g' /etc/mkinitcpio.conf

# Add the new keyfile to FILES for mkinitCPIO
sed -i 's!FILES=()!FILES=(/root/cryptlvm.keyfile)!g' /etc/mkinitcpio.conf

echo "mkinitcpio.conf has been updated"

#read -p "Press any key to continue..." -n1 -s

# Create a new initramfs
mkinitcpio -P

# Change permissions on intrafs files after mkinitcpio runs
chmod 600 /boot/initramfs-linux*

# Update sudo
echo "%wheel ALL=(ALL:ALL) ALL" >> /etc/sudoers.d/01_wheel
echo "aur_installer ALL=(ALL) NOPASSWD: /usr/bin/pacman" >> /etc/sudoers.d/99_aur_installer
echo "aur_installer ALL=(ALL) NOPASSWD: /usr/bin/aur" >> /etc/sudoers.d/99_aur_installer

# Enable the NetworkManager service so that an IP will be assigned after reboot
systemctl enable NetworkManager

# Enable SSHD (server)
systemctl enable sshd

# Find the system drive again
SYSTEM_DRIVE="$(fdisk -l | grep 'GiB' | grep -v mapper | awk '{print $2}' | sed 's/.$//')"

# Configure default grub configs
sed -i "s!GRUB_CMDLINE_LINUX=\"\"!GRUB_CMDLINE_LINUX=\"cryptdevice=${SYSTEM_DRIVE}2:lvmvirt root=/dev/mapper/vglinux-lvlinux cryptkey=rootfs:/root/cryptlvm.keyfile\"!g" /etc/default/grub
sed -i '/GRUB_ENABLE_CRYPTODISK=y/s/^#//g' /etc/default/grub

#read -p "Press any key to continue..." -n1 -s

# Setup grub
grub-install --target=x86_64-efi --bootloader-id=ARCH --efi-directory=/boot/efi

# Configure grub boot menu
grub-mkconfig -o /boot/grub/grub.cfg

# Exit out of Chroot
#exit

######### Success Exit code for the script #########
#exit 0

During the script after it runs the grub-install I receive this output:

[root@archiso /]# grub-install --bootloader-id=Arch --efi-directory=/boot/efi
Installing for x86_64-efi platform.
grub-install: error: unknown filesystem.

Based on the requests from other forum posts, here is the output to a bunch of different commands:

[root@archiso /]# lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                   7:0    0 757.8M  1 loop  
sr0                    11:0    1 883.3M  0 rom   
vda                   254:0    0    25G  0 disk  
├─vda1                254:1    0   512M  0 part  /boot/efi
└─vda2                254:2    0  24.5G  0 part  
  └─lvmvirt           253:0    0  24.5G  0 crypt 
    └─vglinux-lvlinux 253:1    0  24.5G  0 lvm   /

[root@archiso /]# fdisk -l /dev/vda
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C2431E30-BB4C-407C-915E-56910DAC6F12

Device       Start      End  Sectors  Size Type
/dev/vda1     2048  1050623  1048576  512M EFI System
/dev/vda2  1050624 52428766 51378143 24.5G Linux filesystem

[root@archiso /]# mount
/dev/mapper/vglinux-lvlinux on / type ext4 (rw,relatime)
/dev/vda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=946032k,nr_inodes=236508,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
tmp on /tmp type tmpfs (rw,nosuid,nodev,inode64)
run on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)

root@archiso /]# cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/vglinux-lvlinux
UUID=33559211-da71-4342-b14c-47c34f4578bc	/         	ext4      	rw,relatime	0 1

# /dev/vda1
UUID=609D-B14B      	/boot/efi 	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

[root@archiso /]# cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=(/root/cryptlvm.keyfile)

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block encrypt filesystems resume lvm2 fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems resume lvm2 fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"

I have tested all of this without encryption, and it works. I have encryption on my desktop and my laptop, and they work. So I think it has to do with the VM setup some how, but I am not sure what it is. Maybe I am missing some OVMF file that needs to be there for grub to install to efi?
I have tried things like removing the cryptfile from mkinicpio.conf as a test. I have altered the /etc/default/grub a couple of different times, but in all cases the grub-install just errors out.
I had also tried a couple of tunefs commands that I found in other forums. I'm not sure what the commands were supposed to do, but I had tried them anyways.
Also, all of the tests that I ran, were after I deleted the VM and recreated it. So it shouldn't be a problem of me trying too many things at once on the same installation. The installation is almost fully automated (minus the new arch-chroot problem), so it only takes a few minutes for me to bring up a new VM to test with.
I have been working on this for a couple of days now without success, so I have tried a bunch of things, which I can begin to post details if need be.

******* Edit ********
Just in case it matters, here is the info that I use for the EFI on Proxmox

bios = "ovmf"
machine = "q35"
efi_config {
efi_storage_pool = "l1"
efi_type = "4m"
pre_enrolled_keys = false
}

All of the install is handled by Packer and Ansible.
Any suggestions on something else I could or should check?

Last edited by GuloGuloDesu (2024-02-01 17:54:41)

kermit63

Member

Registered: 2018-07-04

Posts: 174

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.

GuloGuloDesu

Member

Registered: 2013-07-18

Posts: 35

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.

That is correct, the /boot directory is under the encrypted root directory. The /boot/efi directory is not, during the boot process, it will ask for a decrypt password to unlock /dev/mapper/vglinux-lvlinux, after that is unlocked grub will start and then ask for the password to unlock /dev/mapper/lvmvirt unless the cryptlvm.keyfile is configured in grub.

Here is the output from my laptop that I rebuilt this weekend:

10:18:52 ~/ $ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1               259:0    0 953.9G  0 disk  
├─nvme0n1p1           259:1    0   511M  0 part  /boot/efi
└─nvme0n1p2           259:2    0 953.4G  0 part  
  └─lvmnvme           254:0    0 953.4G  0 crypt 
    ├─vglinux-lvswap  254:1    0    50G  0 lvm   [SWAP]
    └─vglinux-lvlinux 254:2    0 903.4G  0 lvm   /

0:19:55 ~/ $ sudo fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: WDC PC SN730 SDBPNTY-1T00               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 48AA2B3E-C681-445A-94B4-0C107FEF2915

Device           Start        End    Sectors   Size Type
/dev/nvme0n1p1    2048    1048576    1046529   511M EFI System
/dev/nvme0n1p2 1050624 2000409230 1999358607 953.4G Linux filesystem

It also has the /boot drive under the encrypted drive, but /boot/efi is unencrypted.

But that is something that I did not test. I will change the script a bit and build a new VM with /boot being directly mounted to /dev/vda1 so that it is unencrypted and see what happens.

GuloGuloDesu

Member

Registered: 2013-07-18

Posts: 35

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

Alright, so changing the mounting of /dev/vda1 to /mnt/boot rather than /mnt/boot/efi solves the problem.

I am not sure how I feel about not having the /boot unecrypted as well. I will definitely need to remove the /cryptlvm.keyfile from GRUB so that it asks for the password on boot, rather than just unlocking.

So this could be a workaround if I am unable to get the /boot partition encrypted as well.

GuloGuloDesu

Member

Registered: 2013-07-18

Posts: 35

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.

Rereading your post, I just realized that you said luks2.
The cryptsetup command that I am using switches the type to luks1 to avoid the incompatibility with luks2. That was a fun problem to debug a couple of weeks ago. So grub should / is compatible with the setup that I am using (works on desktop and laptop).

kermit63

Member

Registered: 2018-07-04

Posts: 174

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

I stand corrected then.

EDIT:  It may be nothing, but I noticed that in your mkinitcpio.conf hook line:

HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems resume lvm2 fsck)

The lvm2 module come in very late. Sometimes the order of the modules matter. Maybe try putting it between block and encrypt and see if it makes a difference.

Last edited by kermit63 (2024-02-02 07:37:32)

GuloGuloDesu

Member

Registered: 2013-07-18

Posts: 35

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

I stand corrected then.

The lvm2 module come in very late. Sometimes the order of the modules matter. Maybe try putting it between block and encrypt and see if it makes a difference.

I tried moving the lvm2 to a few different locations in /etc/mkinitcpio.conf, but that did not make a difference.

I did learn about a new switch for the grub-installer called verbose (ha!). It appears that grub-install is failing because the filesystem type is crypt0

[root@archiso /]# grub-install --target=x86_64-efi --bootloader-id=Arch --efi-directory=/boot/efi --recheck --verbose --no-floppy
grub-install: info: Scanning for lvm devices on disk crypto0.
grub-install: info: Found array vglinux.
grub-install: info: Inserting crypto0 (+0,51374047) into vglinux (lvm)
.
grub-install: error: unknown filesystem.

I also tried using grub-probe /boot and it has the same error.

[root@archiso /]# grub-probe /boot --verbose
grub-probe: info: Scanning for lvm devices on disk crypto0.
grub-probe: info: Found array vglinux.
grub-probe: info: Inserting crypto0 (+0,51374047) into vglinux (lvm)
.
grub-probe: error: unknown filesystem.

Now, when I run the grub-probe on my laptop, I receive this response:

12:26:12 ~/ $ sudo grub-probe --verbose /boot
grub-probe: info: Scanning for lvm devices on disk crypto0.
grub-probe: info: Found array vglinux.
grub-probe: info: Inserting crypto0 (+0,1999354511) into vglinux (lvm)
.
ext2

I can't figure out what is different between the laptop and the VM. I used the same scripts to build both. The only differences are:
A few days, maybe a week between the installs
Physical machine versus a VM

So it is a definite problem with crypt0. There must be some dependency that I am missing somewhere. It could be a config problem, but I don't quite know how it would be.

Once again, I am open to any and all suggestions to try and figure this one out.

Strike0

Member

From: Germany

Registered: 2011-09-05

Posts: 1,429

Re: EFI, LUKS, and LVM grub-install error: unknown filesystem

Looking at the wiki, I notice both the dm-crypt mount (mount /dev/MyVolGroup/root /mnt) and the kernel's (root=/dev/MyVolGroup/root) don't use dashes anymore. It used to not matter, but I'm not sure.
Other than that:
(1) run the grub-install outside the chroot, prepending "--boot-directory=/mnt/boot" before the "--efi-directory=/boot/efi"
(2) alternatively, since you just use one lv, I'd eliminate the lvm completely and see if that changes it.