You are not logged in.

Pages: 1

#1 2020-05-15 14:30:51

post-factum
Member
From: /cz
Registered: 2008-09-12
Posts: 149
Website

[SOLVED] Some LVM thin volumes do not get activated during boot

This bothers me for quite some time, but I've got no idea on how to move forward, so asking for help.

I have an issue with activating some thin LVs during boot. Usually, this happens:

kvě 15 09:31:39 archlinux systemd[1]: Starting LVM2 PV scan on device 254:2...
kvě 15 09:31:39 archlinux lvm[446]:   pvscan[446] VG sys run autoactivation.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:39 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:39 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:40 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:40 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:40 archlinux lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:40 archlinux lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:40 butterfly lvm[446]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 09:31:40 butterfly lvm[446]:   WARNING: Failed to monitor sys/pool.
kvě 15 09:31:40 butterfly lvm[446]:   /run/lock/lvm/A_sys:aux: open failed: No such file or directory
kvě 15 09:32:09 butterfly systemd[1]: lvm2-pvscan@254:2.service: State 'stop-sigterm' timed out. Killing.
kvě 15 09:32:09 butterfly systemd[1]: lvm2-pvscan@254:2.service: Killing process 446 (lvm) with signal SIGKILL.
kvě 15 09:32:10 butterfly systemd[1]: lvm2-pvscan@254:2.service: Main process exited, code=killed, status=9/KILL
kvě 15 09:32:10 butterfly systemd[1]: lvm2-pvscan@254:2.service: Failed with result 'timeout'.
kvě 15 09:32:10 butterfly systemd[1]: Stopped LVM2 PV scan on device 254:2.

and once the boot finishes, some thin LVs are not activated (usually, a couple of the LVs that were created not so long ago). 254:2 refers to the device that is created after opening LUKS LV (/dev/mapper/sys).

I have nested LVM setup with 2 PVs and 2 VGs:

/dev/nvme0n1p2 is the "base" PV/VG that contains boot partition and LUKS partition;
/dev/mapper/sys is the "sys" PV/VG that is created after opening base-sys LUKS partition and contains root, home, swap etc. See on the layout below:

[~]$ lsblk
NAME                                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                      8:0    1   984M  0 disk  
├─sda1                                   8:1    1   656M  0 part  
└─sda2                                   8:2    1    64M  0 part  
nvme0n1                                259:0    0 238,5G  0 disk  
├─nvme0n1p1                            259:1    0     8M  0 part  /boot/EFI
└─nvme0n1p2                            259:2    0 238,5G  0 part  
  ├─base-boot                          254:0    0   256M  0 lvm   /boot
  └─base-sys                           254:1    0 238,2G  0 lvm   
    └─sys                              254:2    0 238,2G  0 crypt 
      ├─sys-swap                       254:3    0     2G  0 lvm   [SWAP]
      ├─sys-pool_tmeta                 254:4    0   120M  0 lvm   
      │ └─sys-pool-tpool               254:6    0   236G  0 lvm   
      │   ├─sys-pool                   254:7    0   236G  1 lvm   
      │   ├─sys-root                   254:8    0   235G  0 lvm   /
      │   ├─sys-home                   254:9    0   235G  0 lvm   /home
      │   ├─sys-containers             254:10   0   235G  0 lvm   /mnt/containers
      │   ├─sys-vms                    254:11   0   235G  0 lvm   /mnt/vms
      │   ├─sys-email                  254:12   0   235G  0 lvm   /mnt/email
      │   ├─sys-vm--01--rhel6--sda     254:13   0     8G  0 lvm   
      │   ├─sys-vm--02--rhel7--sda     254:14   0     8G  0 lvm   
      │   ├─sys-vm--03--rhel8--sda     254:15   0     8G  0 lvm   
      │   ├─sys-vm--04--rhel6--32--sda 254:16   0     8G  0 lvm   
      │   ├─sys-vm--05--rhel5--sda     254:17   0     8G  0 lvm   
      │   ├─sys-vm--06--fedora--sda    254:18   0     8G  0 lvm   
      │   └─sys-vm--07--archlinux--sda 254:19   0     8G  0 lvm   
      └─sys-pool_tdata                 254:5    0   236G  0 lvm   
        └─sys-pool-tpool               254:6    0   236G  0 lvm   
          ├─sys-pool                   254:7    0   236G  1 lvm   
          ├─sys-root                   254:8    0   235G  0 lvm   /
          ├─sys-home                   254:9    0   235G  0 lvm   /home
          ├─sys-containers             254:10   0   235G  0 lvm   /mnt/containers
          ├─sys-vms                    254:11   0   235G  0 lvm   /mnt/vms
          ├─sys-email                  254:12   0   235G  0 lvm   /mnt/email
          ├─sys-vm--01--rhel6--sda     254:13   0     8G  0 lvm   
          ├─sys-vm--02--rhel7--sda     254:14   0     8G  0 lvm   
          ├─sys-vm--03--rhel8--sda     254:15   0     8G  0 lvm   
          ├─sys-vm--04--rhel6--32--sda 254:16   0     8G  0 lvm   
          ├─sys-vm--05--rhel5--sda     254:17   0     8G  0 lvm   
          ├─sys-vm--06--fedora--sda    254:18   0     8G  0 lvm   
          └─sys-vm--07--archlinux--sda 254:19   0     8G  0 lvm

or:

[~]$ sudo pvs
  PV              VG   Fmt  Attr PSize    PFree
  /dev/mapper/sys sys  lvm2 a--  <238,20g    0 
  /dev/nvme0n1p2  base lvm2 a--   238,46g    0 

[~]$ sudo vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  base   1   2   0 wz--n-  238,46g    0 
  sys    1  14   0 wz--n- <238,20g    0 

[~]$ sudo lvs
  LV                  VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  boot                base -wi-ao---- 256,00m                                                    
  sys                 base -wi-ao---- 238,21g                                                    
  containers          sys  Vwi-aotz-- 235,00g pool        1,42                                   
  email               sys  Vwi-aotz-- 235,00g pool        2,76                                   
  home                sys  Vwi-aotz-- 235,00g pool        25,00                                  
  pool                sys  twi-aotz-- 235,96g             50,39  33,69                           
  root                sys  Vwi-aotz-- 235,00g pool        8,26                                   
  swap                sys  -wi-ao----   2,00g                                                    
  vm-01-rhel6-sda     sys  Vwi-a-tz--   8,00g pool        51,39                                  
  vm-02-rhel7-sda     sys  Vwi-a-tz--   8,00g pool        66,37                                  
  vm-03-rhel8-sda     sys  Vwi-a-tz--   8,00g pool        29,91                                  
  vm-04-rhel6-32-sda  sys  Vwi-a-tz--   8,00g pool        25,70                                  
  vm-05-rhel5-sda     sys  Vwi-a-tz--   8,00g pool        75,00                                  
  vm-06-fedora-sda    sys  Vwi-a-tz--   8,00g pool        25,00                                  
  vm-07-archlinux-sda sys  Vwi-a-tz--   8,00g pool        29,79                                  
  vms                 sys  Vwi-aotz-- 235,00g pool        2,84

First, I've tried to get rid of dmeventd errors with this is /etc/mkinitcpio.conf:

BINARIES=(/usr/bin/dmeventd)
FILES=(/usr/lib/libdevmapper-event-lvm2.so /usr/lib/libdevmapper-event-lvm2thin.so /usr/lib/liblvm2cmd.so.2.02)

# also pasting the list of hooks for completeness
HOOKS=(base systemd autodetect modconf block sd-lvm2 sd-encrypt filesystems keyboard fsck)

But this didn't help, leaving me with the following printout instead:

kvě 15 15:48:44 archlinux systemd[1]: Starting LVM2 PV scan on device 254:2...
kvě 15 15:48:44 archlinux lvm[453]:   pvscan[453] VG sys run autoactivation.
kvě 15 15:48:44 archlinux dmeventd[486]: dmeventd ready for processing.
kvě 15 15:48:44 archlinux lvm[486]: Monitoring thin pool sys-pool-tpool.
kvě 15 15:48:44 archlinux lvm[486]: dmeventd received break, scheduling exit.
kvě 15 15:48:45 butterfly lvm[453]:   /usr/bin/dmeventd: stat failed: No such file or directory
kvě 15 15:48:45 butterfly lvm[453]:   WARNING: Failed to monitor sys/pool.
kvě 15 15:48:45 butterfly lvm[453]:   /run/lock/lvm/A_sys:aux: open failed: No such file or directory
kvě 15 15:48:45 butterfly lvm[453]:   /run/lock/lvm/A_sys:aux: open failed: No such file or directory
kvě 15 15:48:45 butterfly lvm[453]:   /run/lock/lvm/A_sys:aux: open failed: No such file or directory
kvě 15 15:48:45 butterfly lvm[486]: dmeventd received break, scheduling exit.
kvě 15 15:48:54 butterfly lvm[486]: WARNING: proc dir /proc not found - some checks will be bypassed
kvě 15 15:48:54 butterfly lvm[486]: Ignoring /dev: No such file or directory.
kvě 15 15:48:54 butterfly lvm[486]: WARNING: Failed to connect to lvmetad. Falling back to device scanning.
kvě 15 15:48:54 butterfly lvm[486]: Volume group "sys" not found
kvě 15 15:48:54 butterfly lvm[486]: Cannot process volume group sys
kvě 15 15:48:54 butterfly lvm[486]: Failed command for sys-pool-tpool.
kvě 15 15:49:14 butterfly lvm[486]: Volume group "sys" not found
kvě 15 15:49:14 butterfly lvm[486]: Cannot process volume group sys
kvě 15 15:49:14 butterfly lvm[486]: Failed command for sys-pool-tpool.
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: State 'stop-sigterm' timed out. Killing.
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: Killing process 453 (lvm) with signal SIGKILL.
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: Killing process 486 (dmeventd) with signal SIGKILL.
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: Killing process 490 (dmeventd) with signal SIGKILL.
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: Main process exited, code=killed, status=9/KILL
kvě 15 15:49:15 butterfly systemd[1]: lvm2-pvscan@254:2.service: Failed with result 'timeout'.
kvě 15 15:49:15 butterfly systemd[1]: Stopped LVM2 PV scan on device 254:2.

I can activate thin LVs later after the boot by randomly restarting lvm2-lvmetad.service, lvm2-pvscan@254:2.service and lvm2-monitor.service, but in that case no symlinks under /dev/sys (where "sys" is the name of VG that resides on LUKS LV) are created for the volumes that were activated during the boot, they are created only for those that are activated after the boot. The symlinks under /dev/mapper appear, though.

Also, I've tried to boot with use_lvmetad=0 with no luck (as I've found later, Arch cannot deal with it now). Shuffling filter/global_filter in the lvm.conf doesn't help much as well, currently I have this there:

filter = [ "a|^/dev/nvme0n1p2$|", "a|^/dev/mapper/sys$|", "r|.*/|" ]
global_filter = [ "a|^/dev/nvme0n1p2$|", "a|^/dev/mapper/sys$|", "r|.*/|" ]

I've stumbled across various lvm2 bugs that are similar to what I experience, but it seems they are either solved by now or just hang in the air without any explanation.

Finally, previously, when I had less thin LVs, the issue popped up not so often, but it was still there.

Any suggestions with this issue please?

Last edited by post-factum (2020-05-16 19:36:08)

uname == latest pf-kernel

Offline

#2 2020-05-15 19:26:58

frostschutz
Member
Registered: 2013-11-15
Posts: 1,417

Re: [SOLVED] Some LVM thin volumes do not get activated during boot

LVM on LVM plus thin pool... a bit exotic :-) try without those filters?

regarding lvmetad there's this https://bugs.archlinux.org/task/57275#comment167331

dmeventd binary missing and dm-event service as well... so no thin pool support in mkinitcpio, plus it seems getting killed on top of that (during switch root?)

just for kicks you could try dracut instead of the initcpio

Offline

#3 2020-05-15 20:31:49

post-factum
Member
From: /cz
Registered: 2008-09-12
Posts: 149
Website

Re: [SOLVED] Some LVM thin volumes do not get activated during boot

frostschutz wrote:

LVM on LVM plus thin pool... a bit exotic :-) try without those filters?

Yeah, exotic, but serves its purpose… Anyway, the filters doesn't make any difference, the same happens with and without them. I need those filters to prevent LVM from scanning LVs dedicated to VMs and leaking guest VGs into the host.

frostschutz wrote:

Hmm… I've applied that lvm2-fix hack, and the system survived 4 reboots in a row. I'll give it more time to be tested, but considering this unit ordering stuff to be a real fix, should this be reported as a bug and fixed directly in the lvm2-lvmetad.service in the stock package?..

frostschutz wrote:

dmeventd binary missing and dm-event service as well... so no thin pool support in mkinitcpio, plus it seems getting killed on top of that (during switch root?)

I'm still not convinced those dmevent messages are relevant since thin LVs are discovered anyway.

frostschutz wrote:

just for kicks you could try dracut instead of the initcpio

Okay, I'll reserve this step for future investigation if it appears that the lvm2-fix hack doesn't work.

Thank you for your help.

uname == latest pf-kernel

Offline

#4 2020-05-15 20:37:28

frostschutz
Member
Registered: 2013-11-15
Posts: 1,417

Re: [SOLVED] Some LVM thin volumes do not get activated during boot

post-factum wrote:

should this be reported as a bug and fixed directly in the lvm2-lvmetad.service in the stock package?

lvmetad got fixed... it got axed entirely.

archlinux is using old version. in ubuntu (19.10 & 20.04) it's already gone.

Last edited by frostschutz (2020-05-15 21:51:22)

Offline

#5 2020-05-16 19:35:51

post-factum
Member
From: /cz
Registered: 2008-09-12
Posts: 149
Website

Re: [SOLVED] Some LVM thin volumes do not get activated during boot

frostschutz wrote:

lvmetad got fixed... it got axed entirely.

archlinux is using old version. in ubuntu (19.10 & 20.04) it's already gone.

OK, I've tested the system with more reboots, and it seems it works with that ordering fix. So, I've opened a bugreport [1], lets see what devs say.

Marking this as solved for now. Thanks.

[1] https://bugs.archlinux.org/task/66679

uname == latest pf-kernel

Offline

Pages: 1

Board footer

Powered by FluxBB