LVM Snapshots to safeguard pacman system upgrade

stevepa · 2015-03-18 21:09:24

I'm a long time LVM user but never really explored the snapshot capability.

I am considering the possibility of using a snapshot of my lvm backed root filesystem, before pacman -Syu actions. So, I can recover if the upgrade fails.
(I currently use tar to do complete file backups that this might replace)

So, if my root filesystem is mounted like this

/dev/mapper/vg05-lvoli5root on / type ext4 (rw,relatime,data=ordered)

And, my boot partition is mounted like this

/dev/sda1 on /boot type ext2 (rw,relatime)

And I backup my boot partition like this:

tar cvf /root/bootbu.tar boot/

I snapshot my root filesystem like this:

lvcreate --size 1G --snapshot --name snaps_root_fs /dev/mapper/vg05-lvoli5root

Update: Note that snapshots must not be named "snapshot..." because of creation error.

Then I do the potentially unrecoverable upgrade

pacman -Syu

Then discover, for whatever reason, I want to revert, so i do

lvconvert --merge /dev/vg05/snaps_root_fs

After roboot, should the snapshot state be restored as if the pacman upgrade never happened?

(I realize that I may need to chroot in and restore the bootbu.tar file to /boot and redo grub...)

Tips or pointers to other documentation appreciated.
Tks

Last edited by stevepa (2015-03-20 21:55:37)

lucke · 2015-03-18 22:55:07

It should work the way you imagine.

Be advised that having snapshots on apparently heavily degrades write performance.

Last edited by lucke (2015-03-18 22:56:19)

frostschutz · 2015-03-18 23:59:41

It rolls back everything, not just the pacman -Syu, so be advised you have no databases or otherwise important data (maildirs, whatever) on the same filesystem. Which is actually quite impossible since you need at least the pacman db on it, right?

There is no way to undo the undo, either...

I haven't used snapshots much myself because it's really difficult to tell what will happen. I do not enjoy travelling back in time in a filesystem. I'd rather take my chances being able to fix whatever broke. The snapshot is a last resort only.

Last edited by frostschutz (2015-03-19 00:01:20)

stevepa · 2015-03-19 04:22:14

@lucke

I will check performance. I am thinking that I won't have the snapshot "on" for more than a few hours, or just enough to ensure the upgrade was not catastrophic.

@frostschutz : That thought crossed my mind. I only do web mail, but can you elaborate about the pacman db problem? Not sure I understand.

I will update once I try it out for real.

Tks to all.
Steve.

Update 1: confirmed that method outlined above completed without error and the snapshot restored system. Will update with performance data next.

Last edited by stevepa (2015-03-20 21:56:46)

DoctorT · 2015-08-04 08:23:11

Hi @stevepa,

Any news on the performance impact?

Also, a question. Are you keeping your snapshot on all the time? Or just before upgrades?

I was thinking it would be more useful to simply create the snapshot before upgrading, performing the upgrade, confirm that everything works fine and eventually use the snapshot to rollback, and then remove the snapshot.

Is there any advantage to keeping the snapshot on all the time?

jasonwryan · 2015-08-04 09:26:55

frostschutz wrote:

I do not enjoy travelling back in time in a filesystem. I'd rather take my chances being able to fix whatever broke.

This.

Back up your data and fix any issues as they arise...

alphaniner · 2015-08-04 19:00:35

I implemented this many years ago with / /var and /home on separate LVs and took snapshots of each before kernel updates and such. I update via chroot from a separate, minimal install, and even wrote a script to do all the tedious bits for me (like manually backing up the kernel and initrd, because /boot is on an MBR partition shared by the minimal and primary installs). I don't recall any notable performance impact, but in my original setup the VG consisted of two PVs; the origin LVs were on the first PV and snapshots always got put on the second PV. Now my PV is an mdadm RAID 10 with four disks.

But for several years now (even before moving to PV on mdadm), the system intermittently fails to boot when the snapshots exist. The first boot always works; it seems like the odds of epic failure increase with each boot. There's no permanant damage AFAICT; even after a string of failures the system will eventually boot with the snapshots around. Usually I just boot into the minimal system and delete them after the first failure. I've never found a solution, or gotten any input as to what may be going wrong. OTOH I only know of one or two others who have had the same issue.

greyseal96 · 2015-10-10 01:16:39

I always take snapshots before I update just in case something goes wrong and I need to get back quick to a known working good configuration. If I can, I usually try to fix the cause of any problems, but there have been times where I've had to get my computer back to a working state quickly and the snapshots have been a life and time saver. Based on my usage pattern and my experiences this has been a really helpful tool for me.

I have logical volumes for my /, /boot, and /home directories and I have a script that I use to quickly take snapshots of a specified size for some specified logical volumes. I always snapshot all three so that everything is consistent. Having a snapshot of boot taken at the same time as root is essential when doing kernel updates and the like because you can't roll one back without the other. After I've verified that everything's working normally, I drop the snapshots. If there are problems, and I need to roll back quickly, it's super easy to either boot into into the rescue.target using the kernel parameter systemd.unit=rescue.target or to use an Arch Live CD and roll back (merge) the snapshot. I think that the key thing to remember is, as @frostschutz said, it rolls back _everything_. As long as you plan for that by making sure that everything is in a consistent state when you take the snapshots and snapshotting all of the "moving parts" together (like the boot and root volumes), you should be OK. You'd obviously have to have a plan for saving anything on the volume that you don't want rolled back. The couple times I've had to roll back, I just rolled back /boot and / and left my home directory safe with all of my files saved. It works for my usage pattern.

There have been times when I've run with the snapshots for a week or two and I haven't noticed any poor write performance. Then again, I'm not doing write intensive operations on my laptop; only programming. I can second @alphaniner's observation about the boot failures when using the snapshots. When I first started using them, everything seemed to go smoothly. Then I had a couple weird issues. One time, I created some files, worked on them, shut down the computer and then came back the next day and the files weren't there. I recreated the files, worked on them again, and shut down when I finished work. The next day, when I started up, the old files were back again and the new files that I'd (re)created were nowhere to be found. I finally figured out that it was because my fstab was using the UUIDs of the logical volumes to mount the LVs. That's normally great but, since snapshots are _exact_replicas_ of the source LV, they have the same UUID as the source LV. During boot, the system just grabs the first LV with a matching UUID. I finally figured out that the normal volumes had been mounted the first time the files were written, the second time the snapshots got mounted, and the third time, the normal volume got mounted again. I changed my fstab to use the name of the LV, rather than the UUID, and I haven't had that problem anymore. That doesn't mean that my boots are always problem free, though. It's not unusual for Arch not to boot straight away because it can't find the UUID of the root partition. It drops me to a rescue console but I usually just type 'exit' and it continues booting and grabs the correct partitions. I think there's something in the GRUB config that's still calling out the UUID but I'm too lazy to fix the problem and it's not really harming anything. :-) It only rarely affects me. Also, sometimes when I shut down, there's some systemd watchdog service or timer (I can't recall the exact name off the top of my head) that gets stuck. Systemd usually just terminates it after something like 90 seconds and the shut down finishes normally. For me, these are minor inconveniences for the huge benefit of having the safety net of the snapshots to fall back on if I really need them.

As with all things, YMMV. I just thought that I'd contribute my experience with using the snapshots as a safety net for those times when there are problem updates.

Arch Linux

#1 2015-03-18 21:09:24

LVM Snapshots to safeguard pacman system upgrade

#2 2015-03-18 22:55:07

Re: LVM Snapshots to safeguard pacman system upgrade

#3 2015-03-18 23:59:41

Re: LVM Snapshots to safeguard pacman system upgrade

#4 2015-03-19 04:22:14

Re: LVM Snapshots to safeguard pacman system upgrade

#5 2015-08-04 08:23:11

Re: LVM Snapshots to safeguard pacman system upgrade

#6 2015-08-04 09:26:55

Re: LVM Snapshots to safeguard pacman system upgrade

#7 2015-08-04 19:00:35

Re: LVM Snapshots to safeguard pacman system upgrade

#8 2015-10-10 01:16:39

Re: LVM Snapshots to safeguard pacman system upgrade

Board footer