You are not logged in.

#1 2019-08-30 16:10:03

kajims
Member
Registered: 2019-08-30
Posts: 4

Cannot merge root filesystem LVM snapshot at boot

I've tried posting in System Administration but I'm not allowed since I am a new user.

I am trying to setup a test and recovery procedure on a new system, and I want it to be as automatized as possible.

I've followed the wiki for doing this with lvm2 snapshots of the root filesystem:

https://wiki.archlinux.org/index.php/Cr … s_with_LVM

Some instructions seem outdated, so I've used

/usr/lib/systemd/system/dm-event.service

in place of

/usr/lib/systemd/system/dmeventd.service (which is not in my system)

and similarly I've enabled lvm2-monitor.service in place of lvm-monitoring.service

For the rest I did everything as in the wiki. When I boot with the custom GRUB entry, the snapshot is successfully created. However when I issue the command to merge the snapshot back with its origin, and then reboot with the default entry, the system hangs indefinitely at boot. I'm running LVM on LUKS.
At the hanging boot, I get these messages:

(1 of 2) A start job is running for Monitoring of LVM2 mirr sing dmeventd or progress polling ( <time count> / no limit)
(2 of 2) A start job is running for LVM2 PV scan on device 254:0 ( <time count> / no limit)

I've tried reinstalling arch with MODULES=(dm_mod dm-snapshot) in /etc/mkinitcpio.conf , with and without enabling lvm2-monitor.service . I've also tried creating a temporary LV root snapshot during arch installation, building the initramfs image while this snapshot was active and then remove it at the end of installation,
all of these result in hanging boot after issuing the merge command.

This has taken me a lot of time already, so if there is no fix to lvm2 root snapshots what other way of achieving the same goal would you suggest?

Offline

#2 2019-08-30 17:10:09

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: Cannot merge root filesystem LVM snapshot at boot

kajims wrote:

I've tried posting in System Administration but I'm not allowed since I am a new user.

https://bbs.archlinux.org/viewtopic.php?id=218234

Offline

#3 2019-09-02 18:15:29

Jebtrix
Member
Registered: 2016-08-10
Posts: 8

Re: Cannot merge root filesystem LVM snapshot at boot

I have just experienced the same exact problem. I'm not using LUKS however. I wanted to test out LVM2 functionality without any added failure surface. My particular setup is only /root as a logical volume 50GB of  100GB PV/VG on same single physical drive (/home /boot are non lvm). I have not automated any snapshot creation. It was initiated manually just as a test. When I rebooted after merge it got stuck for 25mins before I just forced a reboot. Running nvme drives with a simple 50MB worth of snapshot data to merge could not possibly take that long.

Despite the stalled first reboot it was successfully merged when I forced a reboot and it booted up normally. I am still digging into this but I thought I would at least throw up a post to +1 this problem.

Offline

#4 2019-09-03 19:44:29

kajims
Member
Registered: 2019-08-30
Posts: 4

Re: Cannot merge root filesystem LVM snapshot at boot

That would mean that the issue is more general than I've thought. It seems to be an old bug that none has ever fixed. Are you using lvm2 or sd-lvm2 as initramfs hook?

Offline

#5 2019-09-04 00:33:03

Jebtrix
Member
Registered: 2016-08-10
Posts: 8

Re: Cannot merge root filesystem LVM snapshot at boot

I got it working with going all team systemd in my hooks. We're already waist deep with systemd mine as well go neck deep wink

Original Not Working:
HOOKS="base udev autodetect keyboard keymap consolefont modconf block lvm2 filesystems fsck"

Working:
HOOKS="base systemd autodetect keyboard sd-vconsole modconf block sd-lvm2 filesystems fsck"

Only other custom value I used in mkinitcpio.conf is nvme module and to be thorough using grub2 with efi.

If you look at /usr/lib/initcpio/install/sd-lvm2 build hook you can see it pulls needed modules you tried specifying manually:  dm-mod dm-snapshot dm-mirror dm-cache dm-cache-smq dm-thin-pool. It needs systemd hook to be present first or it will throw errors building.

Last edited by Jebtrix (2019-09-04 00:34:19)

Offline

#6 2019-09-04 18:28:23

kajims
Member
Registered: 2019-08-30
Posts: 4

Re: Cannot merge root filesystem LVM snapshot at boot

The /usr/lib/initcpio/install/lvm2 build hook loads those same modules so that should not be the reason. I still would like to have an insight on this, but it's good to know there is at least one workaround! Are your booting times reasonably quick? How long to merge a snapshot after a major (1-2 GB) system upgrade?

Offline

#7 2019-09-05 17:01:14

Jebtrix
Member
Registered: 2016-08-10
Posts: 8

Re: Cannot merge root filesystem LVM snapshot at boot

I'd argue boot time is faster (have to look at logs to confirm) but it certainly wasn't slower. As for LVM merging 1-2GB it takes a couple seconds. Of course that is completely dependent on your I/O hardware and how many inodes are involved in the merge.

Offline

#8 2019-09-11 15:53:08

kajims
Member
Registered: 2019-08-30
Posts: 4

Re: Cannot merge root filesystem LVM snapshot at boot

I've managed to devise a workaround for non-systemd initramfs, though I believe there must be a more proper solution.

What I did is to create a custom lvm2 hook by adding this code to /usr/lib/initcpio/hooks/lvm2 :

# runtime hook to let root snapshots be merged
run_hook() {

  # check that lvm binary is loaded into initramfs
  if ! type lvm &>/dev/null; then 
    echo "WARNING: snapmerge hook: lvm binary not found. Skipping hook" >&2
    return 1
  fi

  if echo lvs -o lv_name,lv_attr|lvm|grep 'root'|grep 'O' >&/dev/null; then
    # also consider using command lvpoll instead of vgchange    
    echo "vgchange -an" | lvm >/dev/null
    echo "vgchange -ay" | lvm >/dev/null
    printf "Merging LVM snapshots"   
    while echo lvs -o lv_name,lv_attr | lvm | grep 'root' | grep 'O'\
    >&/dev/null; do
      printf "."          
      sleep 1
    done
    printf "Done\n"
  fi    
}

What it does is just to issue

vgchange -an
vgchange -ay

to trigger the merging of the snapshot, and then wait until the merging is complete.

Placing the code on a separate runtime hook listed directly after the lvm2 hook did not work, the merging is triggered as expected but then the boot hangs after passing to real init as in the original case.

However I am experiencing merging times in the order of minutes (1-15 mins) rather than seconds. I am testing on an old 2.5'' hdd though. Currently I don't have an ssd at hand to test on.

Additionally, I am getting these warnings when the merging is triggered in the initramfs:

/run/lvm/lvmpolld.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmpolld. Proceeding with polling without using lvmpolld
WARNING: Check global/use_lvmpolld in lvm.conf or the lvmpolld daemon state.

I currently don't know if these warnings are relevant or if taking care of them would be beneficial to merging times.

Update: merging the snapshot manually from a liveCD proved to be not any faster then the above method. As a reference, I've merged a snapshot of a fresh Arch install after installing the gnome group (about 2GB). That took as long as 30 mins.

Last edited by kajims (2019-09-11 20:05:25)

Offline

Board footer

Powered by FluxBB