Booting into a Btrfs snapshot with EFI

Epoxyc · 2019-03-31 13:45:55

Hi!
I have recently reinstalled my system and decided to give a try to Btrfs.
Since creating snapshots is super easy, I was wondering if I could have some mechanisms to recover from a broken system without any external boot media.
I have a separate FAT32 boot partition and I use systemd-boot as bootloader.

I saw some interesting tools like snapper and pacman hooks to create snapshots automatically.
Now, I would like to create new entries in my bootloader menu to be able to mount a snapshot as root instead of the regular subvolume.

After some researches I have found the existence of grub-btrfs which would do perfectly the job if I had a boot partition in the snapshot.
The problem is that mine is FAT32 and cannot be snapshot to stay consistent with the root partition.

I am wondering if a such tool exists or if I can write a hook by myself which would:

Clone the /boot files (initramfs, ucode, vmlinuz, etc) and rename them.
Create a snapshot.
Add a boot entry with the duplicated /boot files that boots on the freshly created subvolume.

Now it should be possible to boot from a created entry. However after an update for example, the entry would still boot the snapshot but using the old /boot files, and the "normal" /boot files would be overwritten.
I thought about making the snapshot read-only, so it stays consistent with the associated files. It is then possible to make a rw snapshot, make it the default / and update /boot.

I would like to know if what I try to do is a good or a dumb idea (since I didn't see many people trying to do it).
If it is relevant, is there any tool available for that?
Have you a better idea on how to do that (maybe playing with mkinitcpio or anything)?

Thank you!

Head_on_a_Stick · 2019-03-31 19:02:58

GRUB doesn't need /boot to be on the EFI system partition so you can move it back to the btrfs filesystem.

Epoxyc · 2019-03-31 20:32:41

It seems to be a good news. However except my EFS, my entire disk is encrypted, I have read somewhere that GRUB can open LUKS and LVM partitions, I will investigate more. Thanks!
EDIT: It seems that GRUB does not support LUKS2, I guess I'll have to wait.

Last edited by Epoxyc (2019-03-31 20:45:38)

gibru · 2019-05-03 12:39:49

Hi!

Did you make any progress in the meantime?

Here's what I currently do:

I update my machine using a little bash script (inspired by a German Youtuber) and it works pretty well, but I'm in the process of setting up a new machine and want to get this done in a cleaner way, for instance with pacman hooks as you mention. Anyways, my current system setup is similar to what you're describing:

separate EFI boot partition
systemd-boot
full disk encryption (LUKS)
btrfs root

I won't be going into any details, but here's roughly what happens: every time I update my system, the script creates a snapshot of root in its current stable situation. Then, a copy of the current vmlinuz and initramfs-linux-img is created as vmlinuz-stable and initramfs-linux-stable.img (instead of cloning /boot as you describe). I have two boot entries for systemd-boot, the current one which I simply named Current and the stable one as Stable. These entries point to their respective Linux kernels on boot (vmlinuz and vmlinuz-stable) so I can boot into the stable snapshot created before a system upgrade, in case something goes wrong (though after 8 years on Arch, I can't complain...). I tested this on multiple occasions and it works pretty well. To restore the updated system (current), I simply boot the stable snapshot, mv the stable snapshot to current and vmlinuz-stable to vmlinuz. Then a reboot into current, which is now the pre-upgrade system. On a side note: couple of months ago I created a boot entry labelled Experimental so I can take a snapshot of my current system, boot into that one and experiment a bit and get rid of it easily afterwards. Bit like messing with VMs and containers, but on my actual system.

Now, I'd like to change a few things and am currently researching a bit. First, pacman hooks seem a better solution than some bash script that I have to launch instead. Second, I've been thinking about using Grub instead of systemd-boot, simply to change things a little bit and learn something new. I haven't figured out how to go about the boot partition, though. Meaning, do I keep creating a copy of the kernel under a different name (i.e. -stable), clone the entire partition or something else entirely?...

Epoxyc · 2019-05-03 16:05:39

Hi!
I have something close to your solution, except that I don't add any boot entry.
I have snapper and snap-pac which provides a hook to create a snapshot before and after each update.
Then I added a hook (found on the internet):

/usr/share/libalpm/hooks/50_bootbackup.hook

[Trigger]
Operation = Upgrade
Operation = Install
Operation = Remove
Type = Package
Target = linux*

[Action]
Depends = rsync
Description = Backing up /boot...
When = PreTransaction
Exec = /usr/bin/rsync -a --delete /boot /.bootbackup

This clones the boot partition into the btrfs filesystem before it is updated.
I set up snapper to only keep the 20 most recent snapshots.

When I installed my system I created a read-only snapshot "recovery-ro" and a writable "recovery" copy of it (with a different fstab to mount root as the recovery subvolume).
I also copied the boot files, like vmlinuz became vmlinuz-recovery, etc and created a systemd-boot entry for it.

If something goes wrong, I have just to boot from the recovery snapshot, restore a previous snapshot as root, restore the corresponding boot files, and it's done.
For me, this solution is completely fine (since restoring the system is a really rare task).

If you use grub, I saw that snap-pac-grub should do the job for you (at least for the boot entries).
If your partition uses LUKS1 (Grub doesn't support LUKS2), then you can have /boot on your btrfs filesystem and use snap-pac-grub (and/or? grub-btrfs) to automatically create entries for you.
systemd-boot doesn't and won't support encrypted boot partitions.

In the case of an unencrypted boot partition I don't see any other solution than having numbered boot files (renamed before update) that match with the corresponding entry.
I also thought about putting an archlinux iso on the unencrypted boot partition and create a boot entry for it as recovey. But my partition si too small and I can always use a USB stick.

I hope I answered yours questions.

gibru · 2019-05-03 16:53:26

Thanks for your answer!

Yes, I did some digging today and found on the Arch Wiki that Grub supports LUKS1. I'm currently playing with a test machine so I want to give it a shot. Keeping ESP and /boot separate should probably remove the need to cp the content of /boot since, at that point, it should be possible to include /boot as part of the / snapshot. However, I'm not sure to what extend that is going to work in practice (the Suse documentation mentions something about rollbacks not being possible for boot loader data).

Did the same as you regarding a "recovery" snapshot after a fresh install - however mine is writeable. Your approach seems cleverer so I will definitely set it up like that on my new installation.

I thought about snapper for a while, but for my needs it's overkill. I just need a pre-upgrade snapshot. Or, what is your idea behind keeping that many snapshots? My data resides on a different disk and I'm not sure yet whether to format it with btrfs. My current installation is pretty old so I'm still using LVM with ext4.

A little off topic: what did you do for Swap? I just saw on the Arch Wiki that btrfs supports Swap files since Linux 5.0. I currently still have a Swap partition, but if Swap files are an option now I'll consider it.

Anyways, I like your approach so if I'm going to fail miserably sounds like a good backup plan!

Epoxyc · 2019-05-03 17:42:53

If you use Grub with encrypted boot, the only choice is to have ESP and boot partition separated.
In order to do what you want, ideally grub installed in your ESP (that is unencrypted) and a unique subvolume for root and boot in your encrypted btrfs filesystem.
Then snapshots would include root and boot (which is part of root), no need to worry about it anymore.
I don't think there is a clean way to even backup your ESP (except a copy of it). But I suspect that an update of grub that breaks your grub install in ESP is very unlikely to happen.

As for the swap, I have LVM on top of my unique encrypted partition. The btrfs and the swap partition are LVM volumes, so I can resize them online if I want.
Yes, I saw that swap files are supported now, however you cannot activate CoW nor Compression and it seems they have to bee fully allocated (no gain of space).
The advantages are that if you don't use LVM, you can still have only one encrypted partition (which means only one passphrase) and you can change the size of the swap file easily.
I think I saw somewhere it was also possible to use only one passphrase to unlock multiple partitions with Grub as well, so it is not a big deal.

If you're planning to go the Grub way, you would be able to update from LUKS1 to LUKS2 later anyway (when it is supported if you want to keep grub).
But downgrading from LUKS2 to LUKS1 was impossible for me.

20 is just an arbitrary number and it is not costly. It happens that I install/uninstall a lot of packages during a day, and I need a fully working computer these last days (thesis defense, job interviews, etc...).
So if I notice something is broken somewhere else later because of a prior update, I like to think I have the ability to quickly rollback 20 snapshots earlier. Maybe a policy to keep them regarding the time an not the number should be better.

Also, I have put pacman cache in its own subvolume, so it is not snapshot.

gibru · 2019-05-06 08:58:21

I took the time for some testing and I'm getting close. In a nutshell, here's what I've tested over the weekend:

two partitions: one efi partition and one non-encrypted system partition including /boot
two partitions: one efi partition and one encrypted system partition including /boot

The non-encrypted system was a breeze to install and I got everything up and running in no time. Automatic system-snapshot detection works as intended with the following packages installed: grub, grub-btrfs and efibootmgr. Simply updating grub after making the snapshot will add a boot entry for the snapshot.

The problem I'm currently facing is with an encrypted system. The system hangs at boot complaining that it can't find the root UUID. Here's what I did to enable encryption:

Setting up encryption:

# cryptsetup --type luks1 luksFormat /dev/sdx2
# cryptsetup open /dev/sdx2 cryptroot

And then continue with the usual:

mkfs.btrfs -L "name" /dev/mapper/cryptroot

The rest is identical to the non-encrypted setup except for accounting for encryption with a properly placed encrypt hook in mkinitcpio.conf as well as adding GRUB_ENABLE_CRYPTODISK=y before installing grub with grub-install.

Booting up the system, a password prompt to decrypt appears, once entered the boot entries appear, system boots up but can't find root complaining:

ERROR: device 'UUID=....." not found. Skipping fsck
mount /new_root: can't find UUID="...."
dropping to emergency shell

I double-checked the UUID is the correct one. Also decided to test by label - same problem. So far couldn't figure it out. Maybe you've got any ideas?

EDIT: Never mind, I figured it out - it was an oversight on my part: forgot the add GRUB_CMDLINE_LINUX="cryptdevice=/dev/sdx2:cryptroot" to /etc/default/grub. Everything working now

Another thing: what btrfs layout do you use? I've got pretty flat one with: root, home, pkg and snapshots. Additionally, I added a subvol for swap to test the swap file and exclude it from system snapshots. Disabling CoW and compression is pretty straightforward, except for disabling compression chattr doesn't seem to work, but instead 'btrfs property set file compression "" ' has to be used.

Last edited by gibru (2019-05-06 09:29:02)

Epoxyc · 2019-05-06 13:53:37

Glad to read it, I wouldn't have been a great help with grub anyway.
I use a flat layout just as you, it is simple and not a big deal to mount everything in the fstab since there's only a few subvolumes.
All with zstd compression, it actually saves half of my root space (and a bit of home) and I have the same/better performances than before: enough, so I didn't investigate more.
I might switch to grub when luks2 is supported and if your setup is working well.

gibru · 2019-05-06 15:04:59

Didn't mention that because I thought it wasn't relevant...regarding zstd compression, this was my first gotcha. I only checked if the kernel supports it - should've also paid attention to the boot loader: turns out GRUB doesn't support zstd. Had go back to lzo.

Yeah a flat layout is really convenient. I've been thinking about keeping /home on the root subvol, though. There are several reasons: for one, I don't keep my data on home, never have. For me, it's just the place for user configuration files. But the reason that really got me thinking about including it in system snapshots is that I'd like to experiment with different desktops and make bootable snapshots per desktop and keep all those extra config files separate. Should I want to remove a desktop it'll be as simple as deleting the snapshot. Speaking about /home, how do you deal with .configs? A quick look at the btrfs gotchas shows:

Files with a lot of random writes can become heavily fragmented (10000+ extents) causing thrashing on HDDs and excessive multi-second spikes of CPU load on systems with an SSD or large amount a RAM. [...]
On desktops this primarily affects application databases (including Firefox and Chromium profiles, GNOME Zeitgeist, Ubuntu Desktop Couch, Banshee, and Evolution's datastore.)
Workarounds include manually defragmenting your home directory using btrfs fi defragment. Auto-defragment (mount option autodefrag) should solve this problem in 3.0.

autodefrag as a mount option doesn't seem to be possible on per subvolume basis so I guess that won't be the solution. At the end of the day I wonder if it actually makes sense to bother at all. After 2 years with a btrfs root I haven't experienced any issues so far - which, of course isn't a reason to neglect it...but to occasionally manually defrag at least seems to be an option.

At any rate, will experiment some more, but looks promising overall.

And btw, good luck with your thesis and job hunt!

Epoxyc · 2019-05-06 15:23:32

Concerning fragmentation I've enabled autodefrag, no problem at all even with torrents, I just disabled CoW on my VM folder.
Running manual defrag duplicates the snapshots, it is something I've always avoided to do.
I don't really deal with .configs files, I save them when I copy my home to an external drive regularly. They should be independent of the system state anyway.
I use etckeeper which is convenient when I don't remember what I touched in /etc (and across installations).
Maybe you can see if there is something like that for the config files of users or move that to a new subvolume and use snapshots (or use git).

Thank you!
If you learn any new interesting cool stuff about this setup, I would be glad to know it.

gibru · 2019-05-06 15:47:00

Epoxyc wrote:

Running manual defrag duplicates the snapshots, it is something I've always avoided to do.

Are you referring to something like this: Note

Actually, I didn't really do any proper research regarding autodefrag. Any reasons to enable it? Reasons not to or things to be careful about? And speaking of, which mount options did you set?

Yeah, I turn off CoW for VMs as well. But for the new system I think that won't be an issue. I'll use two SSDs, a small one for my system with btrfs and a big one for data and VMs which will probably be LVM and depending on what I need I just use the appropriate filesystem.

Sure, will keep you up-to-date if I find anything else, but I guess the interesting things are already behind us

Epoxyc · 2019-05-06 15:58:49

gibru wrote:

Are you referring to something like this: Note

Yes exactly, I wanted to recompress a subvolume once and the snapshots of it were duplicated (no CoW anymore).
My mount options are: rw,noatime,compress=zstd,ssd,autodefrag,space_cache
I have seen somewhere that autodefrag was a good solution to databases/torrents fragmentation, etc. (as long as the files stay relatively small).
No problem with it so far, for use cases of this scale, I don't think there's something to worry about.

Arch Linux

#1 2019-03-31 13:45:55

Booting into a Btrfs snapshot with EFI

#2 2019-03-31 19:02:58

Re: Booting into a Btrfs snapshot with EFI

#3 2019-03-31 20:32:41

Re: Booting into a Btrfs snapshot with EFI

#4 2019-05-03 12:39:49

Re: Booting into a Btrfs snapshot with EFI

#5 2019-05-03 16:05:39

Re: Booting into a Btrfs snapshot with EFI

#6 2019-05-03 16:53:26

Re: Booting into a Btrfs snapshot with EFI

#7 2019-05-03 17:42:53

Re: Booting into a Btrfs snapshot with EFI

#8 2019-05-06 08:58:21

Re: Booting into a Btrfs snapshot with EFI

#9 2019-05-06 13:53:37

Re: Booting into a Btrfs snapshot with EFI

#10 2019-05-06 15:04:59

Re: Booting into a Btrfs snapshot with EFI

#11 2019-05-06 15:23:32

Re: Booting into a Btrfs snapshot with EFI

#12 2019-05-06 15:47:00

Re: Booting into a Btrfs snapshot with EFI

#13 2019-05-06 15:58:49

Re: Booting into a Btrfs snapshot with EFI

Board footer