You are not logged in.
Proxmox VE 8.1.3 (qemu)
Arch ISO from 01/01/2024
I am trying to do a base install of Arch on a VM. I am able to install Arch just fine without using any encryption. But when I go to use LUKS to encrypt everything, whenever I run grub-install, I receive an unknown filesystem error. I have read through a bunch of forum posts about this, and so far none of them have helped.
I manually ran all of these commands on my desktop and on my laptop to do the install, and they are both functioning, so I think it has something to do with something missing on the VM.
I am assuming that the error: unknown filesystem that I receive when I run grub-install is what is preventing the system from booting, as it isn't an error that I receive on my desktop or my laptop. Nor is it an error that I receive when I am not using encryption.
Here is the first script that I am running:
######### Global variable definitions #########
FULL_SCRIPT_PATH="$(realpath "$0")"
SCRIPT_NAME="$(basename "$0")"
DIR_PATH="$(dirname $FULL_SCRIPT_PATH)"
######### Function definitions #########
######### Main Script #########
logger -s -t $SCRIPT_NAME -p local3.info "Script started"
#read -p "Press any key to continue..." -n1 -s
# List the drives that are at least 1 GB in size
SYSTEM_DRIVE="$(fdisk -l | grep 'GiB' | awk '{print $2}' | sed 's/.$//')"
logger -s -t $SCRIPT_NAME -p local3.info "Found the drive ${SYSTEM_DRIVE} to use for setup"
# Delete all partitions to clear
sgdisk --zap-all "${SYSTEM_DRIVE}"
# Create a new EFI boot partition
sgdisk --new=1::+512M --typecode 1:ef00 "${SYSTEM_DRIVE}"
# Create a new Linux partition that is the rest of the drive space
sgdisk --new=2::0 "${SYSTEM_DRIVE}"
logger -s -t $SCRIPT_NAME -p local3.info "Drive has been successfully partitioned"
# Enable cryptsetup
modprobe dm-crypt
# Encrypt the drive
cryptsetup -h SHA512 -i 11000 -c aes-xts-plain -y -s 512 luksFormat --type luks1 "${SYSTEM_DRIVE}2"
# Passwords will be typed by Packer
# Unlock the enctypted drive
cryptsetup open "${SYSTEM_DRIVE}2" lvmvirt
# Password will be typed by Packer
# Create a new physical volume
#pvcreate "${SYSTEM_DRIVE}2"
pvcreate /dev/mapper/lvmvirt
# Create a volume group on the new physical volume
#vgcreate vglinux "${SYSTEM_DRIVE}2"
vgcreate vglinux /dev/mapper/lvmvirt
# Create the logical volume on the new volume group that is on the new physical volume
lvcreate -l 100%FREE -n lvlinux vglinux
# Format the new logical volume on the new volume group that is on the new physical volume
mkfs.ext4 /dev/mapper/vglinux-lvlinux
# Format the first partition drive
mkfs.fat -F32 "${SYSTEM_DRIVE}1"
logger -s -t $SCRIPT_NAME -p local3.info "All physical and logical partitions have been created and formatted"
# Mount the primary partition
mount /dev/mapper/vglinux-lvlinux /mnt
# Make the boot and EFI directories
mkdir -p /mnt/boot/efi
# Mount the physical partition to the EFI directory
mount -t vfat "${SYSTEM_DRIVE}1" /mnt/boot/efi
logger -s -t $SCRIPT_NAME -p local3.info "Directories and mount points created"
# Install Arch packages (linux-firmware should be used on physical systems)
pacstrap -K /mnt base base-devel linux
logger -s -t $SCRIPT_NAME -p local3.info "Installation of Arch apps completed"
# Generate the FStab for the UUID's
genfstab -U /mnt >> /mnt/etc/fstab
logger -s -t $SCRIPT_NAME -p local3.info "fstab created"
cp arch_chroot.sh /mnt/
logger -s -t $SCRIPT_NAME -p local3.info "Starting chroot"
# Chroot into the Arch installation
#arch-chroot /mnt /bin/bash /arch_chroot.sh
arch-chroot /mnt
logger -s -t $SCRIPT_NAME -p local3.info "Chroot has completed"
# Unmount the mount points
#umount -R /mnt
# Reboot the system after installation
#reboot
######### Success Exit code for the script #########
logger -s -t $SCRIPT_NAME -p local3.info "Script completed"
#exit 0
I am having to do something a little different, because for whatever reason, when I add a command after arch-chroot, it will not enter chroot. Even if it is just arch-chroot /mnt ls it will not go into chroot, and just execute ls. Weird problem that I will probably create another post for once I get the rest of the installation working. This script was working for the past several weeks with the command after arch-chroot (I have spun up around a dozen VM's using it so far).
Once I am in chroot, I execute this script:
######### Global variable definitions #########
FULL_SCRIPT_PATH="$(realpath "$0")"
SCRIPT_NAME="$(basename "$0")"
DIR_PATH="$(dirname $FULL_SCRIPT_PATH)"
######### Function definitions #########
######### Main Script #########
#read -p "Press any key to continue..." -n1 -s
echo "arch_chroot scripts has started"
# Add a new users
useradd -m -G wheel user_1
useradd -m -s /sbin/nologin aur_installer
# Set the root password
passwd
# Set the password for the new users
passwd user_1
echo "Users create with passwords"
# Set the Time Zone
ln -sf /usr/share/zoneinfo/US/Pacific /etc/localtime
# Sync the hardware clock to the system clock
hwclock --systohc
# Set localization to en_US.UTF-8
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
echo "Timezones, locales and time has been updated"
# Set the hostname
echo arch01 > /etc/hostname
# Deteremine the processor type to determine the ucode that needs to be installed
PROCESSOR_VENDOR_ID=$(lscpu | grep Vendor | grep -V BIOS)
if [[ $PROCESSOR_VENDOR_ID =~ "AMD" ]]
then
UCODE="amd-ucode"
else
UCODE="intel-ucode"
fi
# Update pacman
pacman -Sy
# Install the EFI boot manager
pacman --noconfirm -S efibootmgr grub lvm2 vim dhclient networkmanager nm-connection-editor openssh git python3 linux-firmware ${UCODE}
# Or amd-ucode Should probably build a check for which one to install
# Create keyfile to avoid second decryption password request
dd bs=512 count=4 if=/dev/random of=/root/cryptlvm.keyfile iflag=fullblock
# Change permissions on the new keyfile
chmod 000 /root/cryptlvm.keyfile
echo "Cryptlvm keyfile has been created"
# Populate vconsole.conf file for font into during bootup
echo "FONT=lat2-16" >> /etc/vconsole.conf
echo "FONT_MAP=8859-2" >> /etc/vconsole.conf
# Add lvm2 to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/fsck/lvm2 &/g' /etc/mkinitcpio.conf
# Add resume to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/lvm2/resume &/g' /etc/mkinitcpio.conf
# Add encrypt to the HOOKS for mkinitcpio
sed -i '/^HOOKS=/ s/filesystems/encrypt &/g' /etc/mkinitcpio.conf
# Add the new keyfile to FILES for mkinitCPIO
sed -i 's!FILES=()!FILES=(/root/cryptlvm.keyfile)!g' /etc/mkinitcpio.conf
echo "mkinitcpio.conf has been updated"
#read -p "Press any key to continue..." -n1 -s
# Create a new initramfs
mkinitcpio -P
# Change permissions on intrafs files after mkinitcpio runs
chmod 600 /boot/initramfs-linux*
# Update sudo
echo "%wheel ALL=(ALL:ALL) ALL" >> /etc/sudoers.d/01_wheel
echo "aur_installer ALL=(ALL) NOPASSWD: /usr/bin/pacman" >> /etc/sudoers.d/99_aur_installer
echo "aur_installer ALL=(ALL) NOPASSWD: /usr/bin/aur" >> /etc/sudoers.d/99_aur_installer
# Enable the NetworkManager service so that an IP will be assigned after reboot
systemctl enable NetworkManager
# Enable SSHD (server)
systemctl enable sshd
# Find the system drive again
SYSTEM_DRIVE="$(fdisk -l | grep 'GiB' | grep -v mapper | awk '{print $2}' | sed 's/.$//')"
# Configure default grub configs
sed -i "s!GRUB_CMDLINE_LINUX=\"\"!GRUB_CMDLINE_LINUX=\"cryptdevice=${SYSTEM_DRIVE}2:lvmvirt root=/dev/mapper/vglinux-lvlinux cryptkey=rootfs:/root/cryptlvm.keyfile\"!g" /etc/default/grub
sed -i '/GRUB_ENABLE_CRYPTODISK=y/s/^#//g' /etc/default/grub
#read -p "Press any key to continue..." -n1 -s
# Setup grub
grub-install --target=x86_64-efi --bootloader-id=ARCH --efi-directory=/boot/efi
# Configure grub boot menu
grub-mkconfig -o /boot/grub/grub.cfg
# Exit out of Chroot
#exit
######### Success Exit code for the script #########
#exit 0
During the script after it runs the grub-install I receive this output:
[root@archiso /]# grub-install --bootloader-id=Arch --efi-directory=/boot/efi
Installing for x86_64-efi platform.
grub-install: error: unknown filesystem.
Based on the requests from other forum posts, here is the output to a bunch of different commands:
[root@archiso /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 757.8M 1 loop
sr0 11:0 1 883.3M 0 rom
vda 254:0 0 25G 0 disk
├─vda1 254:1 0 512M 0 part /boot/efi
└─vda2 254:2 0 24.5G 0 part
└─lvmvirt 253:0 0 24.5G 0 crypt
└─vglinux-lvlinux 253:1 0 24.5G 0 lvm /
[root@archiso /]# fdisk -l /dev/vda
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C2431E30-BB4C-407C-915E-56910DAC6F12
Device Start End Sectors Size Type
/dev/vda1 2048 1050623 1048576 512M EFI System
/dev/vda2 1050624 52428766 51378143 24.5G Linux filesystem
[root@archiso /]# mount
/dev/mapper/vglinux-lvlinux on / type ext4 (rw,relatime)
/dev/vda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=946032k,nr_inodes=236508,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
tmp on /tmp type tmpfs (rw,nosuid,nodev,inode64)
run on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
root@archiso /]# cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.
# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/vglinux-lvlinux
UUID=33559211-da71-4342-b14c-47c34f4578bc / ext4 rw,relatime 0 1
# /dev/vda1
UUID=609D-B14B /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro 0 2
[root@archiso /]# cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run. Advanced users may wish to specify all system modules
# in this array. For instance:
# MODULES=(usbhid xhci_hcd)
MODULES=()
# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image. This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()
# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way. This is useful for config files.
FILES=(/root/cryptlvm.keyfile)
# HOOKS
# This is the most important setting in this file. The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added. Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
## This setup specifies all modules in the MODULES setting above.
## No RAID, lvm2, or encrypted root is needed.
# HOOKS=(base)
#
## This setup will autodetect all modules for your system and should
## work as a sane default
# HOOKS=(base udev autodetect modconf block encrypt filesystems resume lvm2 fsck)
#
## This setup will generate a 'full' image which supports most systems.
## No autodetection is done.
# HOOKS=(base udev modconf block filesystems fsck)
#
## This setup assembles a mdadm array with an encrypted root file system.
## Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
# HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
## This setup loads an lvm2 volume group.
# HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
## NOTE: If you have /usr on a separate partition, you MUST include the
# usr and fsck hooks.
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems resume lvm2 fsck)
# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()
# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"
I have tested all of this without encryption, and it works. I have encryption on my desktop and my laptop, and they work. So I think it has to do with the VM setup some how, but I am not sure what it is. Maybe I am missing some OVMF file that needs to be there for grub to install to efi?
I have tried things like removing the cryptfile from mkinicpio.conf as a test. I have altered the /etc/default/grub a couple of different times, but in all cases the grub-install just errors out.
I had also tried a couple of tunefs commands that I found in other forums. I'm not sure what the commands were supposed to do, but I had tried them anyways.
Also, all of the tests that I ran, were after I deleted the VM and recreated it. So it shouldn't be a problem of me trying too many things at once on the same installation. The installation is almost fully automated (minus the new arch-chroot problem), so it only takes a few minutes for me to bring up a new VM to test with.
I have been working on this for a couple of days now without success, so I have tried a bunch of things, which I can begin to post details if need be.
******* Edit ********
Just in case it matters, here is the info that I use for the EFI on Proxmox
bios = "ovmf"
machine = "q35"
efi_config {
efi_storage_pool = "l1"
efi_type = "4m"
pre_enrolled_keys = false
}
All of the install is handled by Packer and Ansible.
Any suggestions on something else I could or should check?
Last edited by GuloGuloDesu (2024-02-01 17:54:41)
Offline
Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.
Online
Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.
That is correct, the /boot directory is under the encrypted root directory. The /boot/efi directory is not, during the boot process, it will ask for a decrypt password to unlock /dev/mapper/vglinux-lvlinux, after that is unlocked grub will start and then ask for the password to unlock /dev/mapper/lvmvirt unless the cryptlvm.keyfile is configured in grub.
Here is the output from my laptop that I rebuilt this weekend:
10:18:52 ~/ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 953.9G 0 disk
├─nvme0n1p1 259:1 0 511M 0 part /boot/efi
└─nvme0n1p2 259:2 0 953.4G 0 part
└─lvmnvme 254:0 0 953.4G 0 crypt
├─vglinux-lvswap 254:1 0 50G 0 lvm [SWAP]
└─vglinux-lvlinux 254:2 0 903.4G 0 lvm /
0:19:55 ~/ $ sudo fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: WDC PC SN730 SDBPNTY-1T00
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 48AA2B3E-C681-445A-94B4-0C107FEF2915
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1048576 1046529 511M EFI System
/dev/nvme0n1p2 1050624 2000409230 1999358607 953.4G Linux filesystem
It also has the /boot drive under the encrypted drive, but /boot/efi is unencrypted.
But that is something that I did not test. I will change the script a bit and build a new VM with /boot being directly mounted to /dev/vda1 so that it is unencrypted and see what happens.
Offline
Alright, so changing the mounting of /dev/vda1 to /mnt/boot rather than /mnt/boot/efi solves the problem.
I am not sure how I feel about not having the /boot unecrypted as well. I will definitely need to remove the /cryptlvm.keyfile from GRUB so that it asks for the password on boot, rather than just unlocking.
So this could be a workaround if I am unable to get the /boot partition encrypted as well.
Offline
Unless I'm reading your script wrong, your /boot is inside the root directory, which in turn is luks2 encrypted. GRUB can't handle that.
Rereading your post, I just realized that you said luks2.
The cryptsetup command that I am using switches the type to luks1 to avoid the incompatibility with luks2. That was a fun problem to debug a couple of weeks ago. So grub should / is compatible with the setup that I am using (works on desktop and laptop).
Offline
I stand corrected then.
EDIT: It may be nothing, but I noticed that in your mkinitcpio.conf hook line:
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block encrypt filesystems resume lvm2 fsck)
The lvm2 module come in very late. Sometimes the order of the modules matter. Maybe try putting it between block and encrypt and see if it makes a difference.
Last edited by kermit63 (2024-02-02 07:37:32)
Online
I stand corrected then.
The lvm2 module come in very late. Sometimes the order of the modules matter. Maybe try putting it between block and encrypt and see if it makes a difference.
I tried moving the lvm2 to a few different locations in /etc/mkinitcpio.conf, but that did not make a difference.
I did learn about a new switch for the grub-installer called verbose (ha!). It appears that grub-install is failing because the filesystem type is crypt0
[root@archiso /]# grub-install --target=x86_64-efi --bootloader-id=Arch --efi-directory=/boot/efi --recheck --verbose --no-floppy
grub-install: info: Scanning for lvm devices on disk crypto0.
grub-install: info: Found array vglinux.
grub-install: info: Inserting crypto0 (+0,51374047) into vglinux (lvm)
.
grub-install: error: unknown filesystem.
I also tried using grub-probe /boot and it has the same error.
[root@archiso /]# grub-probe /boot --verbose
grub-probe: info: Scanning for lvm devices on disk crypto0.
grub-probe: info: Found array vglinux.
grub-probe: info: Inserting crypto0 (+0,51374047) into vglinux (lvm)
.
grub-probe: error: unknown filesystem.
Now, when I run the grub-probe on my laptop, I receive this response:
12:26:12 ~/ $ sudo grub-probe --verbose /boot
grub-probe: info: Scanning for lvm devices on disk crypto0.
grub-probe: info: Found array vglinux.
grub-probe: info: Inserting crypto0 (+0,1999354511) into vglinux (lvm)
.
ext2
I can't figure out what is different between the laptop and the VM. I used the same scripts to build both. The only differences are:
A few days, maybe a week between the installs
Physical machine versus a VM
So it is a definite problem with crypt0. There must be some dependency that I am missing somewhere. It could be a config problem, but I don't quite know how it would be.
Once again, I am open to any and all suggestions to try and figure this one out.
Offline
Looking at the wiki, I notice both the dm-crypt mount (mount /dev/MyVolGroup/root /mnt) and the kernel's (root=/dev/MyVolGroup/root) don't use dashes anymore. It used to not matter, but I'm not sure.
Other than that:
(1) run the grub-install outside the chroot, prepending "--boot-directory=/mnt/boot" before the "--efi-directory=/boot/efi"
(2) alternatively, since you just use one lv, I'd eliminate the lvm completely and see if that changes it.
Offline