Arch Linux

NadVic17

Member

Registered: 2020-07-03

Posts: 9

Problems with LVM during boot

Hello together,

i have a problem with my LVM setup. The system either doesn't boot at all (only in emergency mode) or just doesn't automatically mount my data partition which is created on LVM.

Here a few details to the setup:
- 2 different disks, one for the system itself, one for additional data-storage
- the root disk is partitioned naturally, while the data-storage uses LVM
- when i use the FSTAB-entry for the LVM-partition, the system doesn't boot
- when i don't use the FSTAB-entry, i can mount the partition manually and usually without any issues
- i use systemd-boot, if it matters
- i don't use encryption in either disk (neither in the system, nor the LVM partition/disk)

/etc/fstab

#/dev/mapper/Datastore-data	/mnt/data 	ext4      	rw,relatime	0 2
#/dev/Datastore/data	/mnt/data 	ext4      	rw,relatime	0 2

/etc/mkinitcpio.conf

MODULES=(dm_mod)
BINARIES=()
FILES=()
#HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck) # i tried this before
HOOKS=(base systemd autodetect modconf kms keyboard keymap consolefont block lvm2 filesystems fsck)
COMPRESSION="zstd"

I inserted this into the /etc/lvm/lvm.conf, but i think it's deprecated:

...
global {
	use_lvmetad = 1
...
}
...

a few journalctl extracts during my boot process

...
Feb 24 15:00:49 test systemd[1]: Finished Load Kernel Module configfs.
Feb 24 15:00:49 test systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
...
Feb 24 15:00:49 test lvm[337]: PV /dev/sdb1 online, VG Datastore is complete.
Feb 24 15:00:49 test lvm[337]: VG Datastore finished
...
# here a timeout of lvm-activate-Datastore happens
...
Feb 24 15:02:18 test systemd[1]: lvm-activate-Datastore.service: State 'stop-sigterm' timed out. Killing.
Feb 24 15:02:18 test systemd[1]: lvm-activate-Datastore.service: Killing process 215 (lvm) with signal SIGKILL.
Feb 24 15:02:19 test systemd[1]: lvm-activate-Datastore.service: Main process exited, code=killed, status=9/KILL
Feb 24 15:02:19 test systemd[1]: lvm-activate-Datastore.service: Failed with result 'timeout'.
Feb 24 15:02:19 test lvm[248]:   1 logical volume(s) in volume group "Datastore" monitored
Feb 24 15:02:19 test systemd[1]: Stopped /usr/bin/lvm vgchange -aay --autoactivation event Datastore.
Feb 24 15:02:19 test systemd[1]: Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Feb 24 15:02:19 test systemd[1]: Reached target Preparation for Local File Systems.
Feb 24 15:02:19 test systemd[1]: Mounting /boot...
Feb 24 15:02:19 test systemd[1]: Virtual Machine and Container Storage (Compatibility) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/machines.raw).
Feb 24 15:02:19 test systemd[1]: Mounted /boot.
...

VGS

  VG        #PV #LV #SN Attr   VSize VFree
  Datastore   1   1   0 wz--n- 1.95t    0

LVS

  LV   VG        Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data Datastore -wi-a----- 1.95t

PVS

  PV         VG        Fmt  Attr PSize PFree
  /dev/sdb1  Datastore lvm2 a--  1.95t    0

systemctl status lvm-activate-Datastore.service

× lvm-activate-Datastore.service - /usr/bin/lvm vgchange -aay --autoactivation event Datastore
     Loaded: loaded (/run/systemd/transient/lvm-activate-Datastore.service; transient)
  Transient: yes
     Active: failed (Result: timeout) since Sat 2024-02-24 15:02:19 CET; 33min ago
   Duration: 933ms
    Process: 215 ExecStart=/usr/bin/lvm vgchange -aay --autoactivation event Datastore (code=killed, signal=KILL)
   Main PID: 215 (code=killed, signal=KILL)
        CPU: 5ms

Feb 24 15:00:48 archlinux systemd[1]: Started /usr/bin/lvm vgchange -aay --autoactivation event Datastore.
Feb 24 15:02:18 test systemd[1]: lvm-activate-Datastore.service: State 'stop-sigterm' timed out. Killing.
Feb 24 15:02:18 test systemd[1]: lvm-activate-Datastore.service: Killing process 215 (lvm) with signal SIGKILL.
Feb 24 15:02:19 test systemd[1]: lvm-activate-Datastore.service: Main process exited, code=killed, status=9/KILL
Feb 24 15:02:19 test systemd[1]: lvm-activate-Datastore.service: Failed with result 'timeout'.
Feb 24 15:02:19 test systemd[1]: Stopped /usr/bin/lvm vgchange -aay --autoactivation event Datastore.

I suppose the fix is rather easy, but i'm not advanced enough to see it. How can i fix my boot-process, so that my LVM gets automatically mounted during boot and without manual interaction.

Can anyone help me out, please?
Thanks in advance!

NadVic17

Member

Registered: 2020-07-03

Posts: 9

Re: Problems with LVM during boot

Also, probably an important information:
i use a different ISO to deactivate the fstab-entry. Afterwards, i can boot the system. So i activate the fstab-entry again, and ... it works. Also, it works across "reboots".
However, when using a normal shutdown, it doesn't work anymore. Then i instead get this on bootup:

A start job is running for /dev/Datastore/data (1:30)
A stop job is running for /usr/bin/lvm vgchange -aay --autoactivation event Datastore
A start job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling

And then i'm thrown into emergency mode.

NadVic17

Member

Registered: 2020-07-03

Posts: 9

Re: Problems with LVM during boot

Short update:
the problem is possibly fixed through a small script that runs as initcpio-hook, which activates the specific device i need. I've found out that lvmetad isn't reliable, as it seems, which matches my results. The workaround was presented in a similar script. I'll let this run for a few days/weeks and come back with a definitive answer, if this is an indeed reliable, even though hacky, fix.

/etc/initcpio/hooks/lvm2

#!/usr/bin/bash
run_hook() {
  lvm vgchange --sysinit -ay /dev/Datastore
}

Also i tried figuring out why the normal "lvm vgchange -ay" doesn't work for every device, but couldn't find out an actual answer. While troubleshooting i also noticed that using the former unspecified command doesn't yield the results i expected, hence i couldn't find any device under "/dev/mapper". However, using it specifically for "/dev/Datastore" works.

If you encounter something similar, please try:

vgscan # use this to figure out your VG name
vgchange -ay "$YOUR_VG_NAME"  # insert your VG name here

Last edited by NadVic17 (2024-02-24 16:43:05)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,419

Re: Problems with LVM during boot

Get rid of the lvmetad in your config. It does not exist anymore (if you're using lvm 2.03+).

The file /etc/initcpio/hooks/lvm2 should not have any effect. Hooks should use ash, not bash. If install/lvm2 does not have add_runscript, this hook won't even be in your initramfs at all (check with lsinitcpio). Also you seem to be using systemd flavor of initcpio. systemd won't even run any runtime hooks, so even if it were in there, it would not be called.

Not sure what else is going on. Normally - it should just work.

Last edited by frostschutz (2024-02-24 17:37:13)

NadVic17

Member

Registered: 2020-07-03

Posts: 9

Re: Problems with LVM during boot

Yeah, i already got rid of the lvmetad. Was only the flag removed of the entire ... module of LVM?
You're right, the script wasn't loaded at all according to lsinitpcio.

I changed a few other things as well, but maybe i've just been getting lucky with my boots again ... The system runs in Proxmox PVE. The root filesystem is thinly provisioned, while the not mounting data-storage isn't. I've changed some boot orders and tried different SCSI-controllers, but after all reverted back to the first settings again. I've found like 1 or 2 other posts connecting this to Proxmox, but at least 10-15 which weren't connected to any virtualization solution.

I'll try to set some "active loggin" for LVM and see where that gets me, i suppose.