[Solved] Setting up backup offsite replication

Omar007 · 2018-09-22 14:05:09

I'm currently running a backup server using Btrfs in RAID-10. To this server several different systems are backing up data.
- 1x Mac using Time Machine, server selected as the target drive, over AFP.
- 3x Windows using Windows File History, server selected as the target drive, over SMB/CIFS.
- 2x Linux using Btrfs snapshots locally (using snapper), then once a day an incremental snapshot is send to the server with Btrfs send/receive (using snap-sync).

The subvolume tree on the server is like the following:

- Btrfs root
--- backups
----- user1 (mac1. Atm stored in a normal folder on this subvolume)
----- user2 (windows1. Atm stored in a normal folder on this subvolume)
----- user3 (windows2. Atm stored in a normal folder on this subvolume)
----- user4 (windows3 + linux1/2. Atm windows is stored in a normal folder on this subvolume)
------- linux1
--------- day1
--------- day2
--------- ...
------- linux2
--------- day1
--------- day2
--------- ...
-- ...

Now I'm trying to find a way to have these backups replicated from the backup server to an offsite system (also a Btrfs RAID-10 setup) but I can't figure out the proper/most efficient way to handle this.

The following options have passed me:
- Use GlusterFS; would most likely work for the Windows and Mac systems but backing up the Linux systems using Btrfs send/receive will not be possible (from my understanding of how GlusterFS works)
- Use Btrfs send/receive; basically creating a snapshot of a backup, but could work? Something along the lines of creating a read-only snapshot of the backup subvolume and every other r/w subvolume and just sending that plus every other already read-only subvolume? But how to know which ones to send and what is already send? Is there software available that already does exactly this? (buttersink seems it might be able to do the sending part for the read-only subvolumes?)
- Use rsync; might work fine for the Mac and Windows backups but very inefficient for the Btrfs snapshots. (e.g. it's unaware of file renames/moves, has to check all the files on both sides)

I would love to get some input on this subject and to hear some advice/perspectives on how to approach this. Or some other solutions/tools I've missed.

Last edited by Omar007 (2018-09-28 15:55:14)

firecat53 · 2018-09-22 14:34:27

I've been using btrbk for this for several months now and it works great. You can also use it for your local snapshots. If ssh isn't an option for the off-site backups, any of a number of other backup programs could be used (restic, duplicity, etc).

Omar007 · 2018-09-22 17:46:55

A quick peek seems to indicate that btrbk archive might do what I need it but it also seems that btrbk archive doesn't really differ in functionality from buttersink. So while it would still require the creation of snapshots of backups, I think going the Btrfs send/receive route is just going to be the way that's going to work best for this after all.
And from what I can tell about how Time Machine and Windows File History save the files, this should not be a big problem (at worst the latest/ongoing backup during a snapshot could end up being corrupt/partially incomplete).
I'll first move those into their own subvolumes as well though so these are isolated as well.

Looking at the other functionality of btrbk, while I can't use any of that for the target use-case, it does seem it may allow me to change from the several different tools on the client machines to a single tool.
So that is something I'll look into for sure and something that definitely gives it a leg up on just deciding to use something like buttersink in addition to all the other stuff.

The other tools you mention aren't really a solution. These are full blown backup tools so those would create a config where you'd end up with backups of backups instead of replicated backups.

I'll see if I can change one of the Linux systems over to btrbk tomorrow

firecat53 · 2018-09-22 20:14:45

Omar007 wrote:

A quick peek seems to indicate that btrbk archive might do what I need it but it also seems that btrbk archive doesn't really differ in functionality from buttersink. So while it would still require the creation of snapshots of backups, I think going the Btrfs send/receive route is just going to be the way that's going to work best for this after all.

Btrbk uses the btrfs send/receive feature. You actually don't even need the archive feature. Btrbk takes a snapshot and then sends it to however many targets (local and/or remote) you have configured. Archive is more for one-off usage when you plug in an external drive periodically. Took me awhile to wrap my head around that. Here's a partial example for my HOME subvolume (sent to both an internal drive and external drive that's always plugged in):

volume /run/mount/data_root
  subvolume HOME
    target send-receive    /run/mount/backup/btrbk_HOME
    target send-receive    /run/mount/backup_ext_1/btrbk_HOME

Omar007 wrote:

The other tools you mention aren't really a solution. These are full blown backup tools so those would create a config where you'd end up with backups of backups instead of replicated backups.

If your initial backups get corrupted, having a 2nd tool doing an independent backup and sending it offsite can save your data. I use restic for this reason, taking a separate full backup and sending it to B2. Btrbk is just for my on-site backups.

Omar007 · 2018-09-23 00:08:06

firecat53 wrote:

Btrbk uses the btrfs send/receive feature. You actually don't even need the archive feature. Btrbk takes a snapshot and then sends it to however many targets (local and/or remote) you have configured. Archive is more for one-off usage when you plug in an external drive periodically. Took me awhile to wrap my head around that. Here's a partial example for my HOME subvolume (sent to both an internal drive and external drive that's always plugged in):
volume /run/mount/data_root
  subvolume HOME
    target send-receive    /run/mount/backup/btrbk_HOME
    target send-receive    /run/mount/backup_ext_1/btrbk_HOME

Hmm ok so instead of a server replication setup you're proposing to abandon that and use a setup where there is no replication at all, just a client backing up to multiple systems. I can't do that though.
While it would work in theory for the 2 Linux systems, it is not a solution for the other systems as something like that can't be set up (easily/workable) on those. And in practice it's also not a solution for the Linux systems due to the complete backup block that would be caused by the bandwidth limitations as it waits for the upload to complete to the offsite server.

So my idea is still to use the archive functionality. I'm thinking a setup along the lines of:

Linux systems -> btrbk -> x snapshots -> x backups on Server | Server -> btrbk archive -> x backups/archives to offsite (probably skip hourly ones due to bandwidth)
Windows/Mac systems -> File History/Time Machine to Server share | Server -> btrbk -> 1 snapshot -> 1 archive to offsite (always latest, as the transferred snapshot isn't the backup, the contents of the snapshot are)

The resulting solution for the non-Linux systems still isn't amazing but should be workable. I'll start experimenting with this tomorrow.

Of course the most amazing would be if you could just use Btrfs send/receive functionality over GlusterFS and have GlusterFS just do it's usual replication stuff (async setup) but sadly that is not the case :'(
(from my understanding on how it works anyway)

firecat53 wrote:

If your initial backups get corrupted, having a 2nd tool doing an independent backup and sending it offsite can save your data. I use restic for this reason, taking a separate full backup and sending it to B2. Btrbk is just for my on-site backups.

If the initial backup gets corrupted I'm f*ed either way as that means the filesystem was already broken when I started xD
And this kind of setup would still mean abandoning replication and setting up the clients to backup to X remote systems with the same problems as mentioned earlier (though except in theory for the backup block as it's now a different tool doing the offsite one).

Were it not for having systems other than Linux, I would honestly not even be having this question haha. I'd just have btrfs send/received all the things

Last edited by Omar007 (2018-09-23 00:16:41)

Omar007 · 2018-09-28 15:55:01

So the system has been running for a week and has been working great so far. It has been set up using a combination of run and archive commands.

The TL;DR:
Windows & Mac systems still backup as before to the on-site server. The server uses `btrbk run` to create a backup of the subvolumes that store those backups with a retention policy of only the latest snapshot/backup and the off-site server as the target.
The Linux systems backup to the on-site server using `btrbk run`. The server syncs these with the off-site server once a day using `btrbk archive`. Each system has their own archive job config.
The servers use `btrbk run` to backup their root filesystem to themselves and each other.

Marking this as solved. For those more interested in how I ended up setting this up, read on
I might make this available as a package at some later date if people are interested.

Going into this I decided I wanted to set this up to run under its own user. As a result I created a few scripts to set up a user for btrbk as well as the required files/permissions.
I did so using sudoers.d, systemd sysusers and tmpfiles.

btrbk.sysusers

u	btrbk	-	"btrbk daemon user"	/var/lib/btrbk	/usr/bin/bash

btrbk.tmpfiles

d	/var/lib/btrbk		0750	btrbk	btrbk	-	-
L	/var/lib/btrbk/.ssh	-	-		-	-	/etc/btrbk/ssh
z	/var/lib/btrbk/.ssh	-	btrbk	btrbk	-	-

d	/etc/btrbk		-	btrbk	btrbk	-	-
z	/etc/btrbk/*		-	btrbk	btrbk	-	-
z	/etc/btrbk/ssh/*	-	btrbk	btrbk	-	-

btrbk.sudoers

# Allow users of the group btrbk to use btrfs without password
%btrbk ALL=NOPASSWD:/usr/bin/btrfs

To have it work with this user by default I also decided to re-implement some of the service files that come with btrbk. It forces the btrbk user and sudo backend system.
These are accompanied by timer units but these have nothing special compared to the original ones other than the @... unit name (and for my personal preference; the timings). My btrbk configs are nothing special other than, because of the @... unit setup, now requiring a group assigned.
I also added a new service+timer for `btrbk archive` that retrieves the required information from a simple environment file (for now, as btrbk configs currently have no support for putting all of the data in it for archive to work with).

btrbk-run@.service

[Unit]
Description=btrbk backup
Documentation=man:btrbk(1)

[Service]
Type=oneshot
User=btrbk
Group=btrbk
ExecStart=/usr/bin/btrbk \
	--override backend=btrfs-progs-sudo \
	--override ssh_user=btrbk \
	%j %i

btrbk-archive@.service

[Unit]
Description=btrbk archive
Documentation=man:btrbk(1)

[Service]
Type=oneshot
User=btrbk
Group=btrbk
EnvironmentFile=/etc/btrbk/%i.archive
ExecStart=/usr/bin/btrbk \
	--override backend=btrfs-progs-sudo \
	--override ssh_user=btrbk \
	--override archive_preserve_min="${PRESERVE_MIN}" \
	--override archive_preserve="${PRESERVE}" \
	%j "${SRC}" "${DST}"

sampleMachine.archive

SRC="/mnt/backups/sampleMachine"
DST="ssh://offsite.example.com/mnt/backups/sampleMachine"
PRESERVE_MIN="latest"
PRESERVE="24h 7d"

I also decided to make the btrbk user a bit more lenient with accepting remote systems (so you don't need to hop into the btrbk user to accept the host keys every time).
I may change this later to always accept new ones but to prevent changes to existing ones (just don't redirect the file to /dev/null for that).

ssh.config

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

After setting this up it's basically enough to exchange the btrbk user's SSH key with the target server(s) and just enable the timers or run the service.

Last edited by Omar007 (2018-09-28 15:59:05)

Arch Linux

#1 2018-09-22 14:05:09

[Solved] Setting up backup offsite replication

#2 2018-09-22 14:34:27

Re: [Solved] Setting up backup offsite replication

#3 2018-09-22 17:46:55

Re: [Solved] Setting up backup offsite replication

#4 2018-09-22 20:14:45

Re: [Solved] Setting up backup offsite replication

#5 2018-09-23 00:08:06

Re: [Solved] Setting up backup offsite replication

#6 2018-09-28 15:55:01

Re: [Solved] Setting up backup offsite replication

Board footer