You are not logged in.

#1 2016-02-13 19:23:36

_pheinrich_
Member
Registered: 2014-05-26
Posts: 53

Backup Solution that handles incremental and rename/move

Hi,

currently I am using a rsync command with hardlinks to backup my data hdd to my backup hdd.

        rsync -av --delete --checksum --exclude=lost+found --link-dest="../last" "${SOURCE}" "${DESTINATION}${NOW}"

        cd ${DESTINATION}
        ln -nsf "${NOW}" "last"

This is working great and I love the simplicity.

But today I got in trouble because I renamed a directory containing > 10G data.
Of course the rsync script is copying the whole data twice. It does not have any reference to the old directory name.

I had a look and found this article https://github.com/dparoli/hrsync
The problem with this idea is that the shadow directory needs the whole space of my data directory before calculating the diff.
And this script will create a 1-to-1 mirror and no incremental tree as my initial script.

Anybody with the same problem?

Offline

#2 2016-02-14 16:25:03

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: Backup Solution that handles incremental and rename/move

If both the live data and the back up use btrfs (I don't recommend it, but I'm sure some do this) you can use the send/receive function.

Offline

#3 2016-02-19 18:01:01

_pheinrich_
Member
Registered: 2014-05-26
Posts: 53

Re: Backup Solution that handles incremental and rename/move

Yes you are right. I have thought of it.

Currently I am playing around a litte bit with two old 80GB drives to understand a litte more how btrfs works.
The neat thing is that it is already build into the kernel and I do not have to worry about compiling a new kernel module for each update (zfs).

But there are not much howtos around there.

I have the following setup:

my data Drive mounted on /mnt/data (dev/sdc)

/mnt/data
|-- .snapshots
|   `-- music
|       |-- last -> /mnt/data/.snapshots/music/music@20160219_183213
|       |-- music@20160219_182110
|       |   |-- mikky.mp3
|       |   `-- twenty.mp3
|       |-- music@20160219_182130
|       |   |-- mikky.mp3
|       |   `-- twenty.mp3
|       |-- music@20160219_182652
|       |   |-- I\ Exhale\ -\ Underworld.mp3
|       |   |-- mikky.mp3
|       |   `-- twenty.mp3
|       |-- music@20160219_183158
|       |   |-- I\ Exhale\ -\ Underworld.mp3
|       |   |-- mikky.mp3
|       |   `-- twenty.mp3
|       `-- music@20160219_183213
|           |-- I\ Exhale\ -\ Underworld.mp3
|           `-- mikky.mp3
|-- music
|   |-- I\ Exhale\ -\ Underworld.mp3
|   `-- mikky.mp3
|-- picture
`-- video
    `-- Hurts\ -\ Stay.mp4

my backup Drive mounted on /mnt/backup (/dev/sdd)

/mnt/backup
`-- music
    |-- last -> /mnt/backup/music/music@20160219_183213
    |-- music@20160219_182110
    |   |-- mikky.mp3
    |   `-- twenty.mp3
    |-- music@20160219_182130
    |   |-- mikky.mp3
    |   `-- twenty.mp3
    |-- music@20160219_182652
    |   |-- I\ Exhale\ -\ Underworld.mp3
    |   |-- mikky.mp3
    |   `-- twenty.mp3
    |-- music@20160219_183158
    |   |-- I\ Exhale\ -\ Underworld.mp3
    |   |-- mikky.mp3
    |   `-- twenty.mp3
    `-- music@20160219_183213
        |-- I\ Exhale\ -\ Underworld.mp3
        `-- mikky.mp3

backup script
backup.sh 36ded5a1-c0c1-4f7a-b1f1-410e0db57fc6 fc45da7b-417c-467a-bcb9-bcf9e6c99581 music

#!/bin/bash

set -e
set -o pipefail

SOURCE_UUID=$1
DESTINATION_UUID=$2

TARGET=$3
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

SOURCE_MOUNTPOINT=$(findmnt -nr -o TARGET -S UUID=$SOURCE_UUID)
DESTINATION_MOUNTPOINT=$(findmnt -nr -o TARGET -S UUID=$DESTINATION_UUID)

# Make sure the path exists
mkdir -p $SOURCE_MOUNTPOINT/.snapshots/$TARGET

# Create the snapshot and sync
/usr/bin/btrfs subvolume snapshot -r $SOURCE_MOUNTPOINT/$TARGET $SOURCE_MOUNTPOINT/.snapshots/$TARGET/$TARGET@$TIMESTAMP
/usr/bin/sync

## Snapshot is taken and will be send now ...

# Make sure the path exists
mkdir -p $DESTINATION_MOUNTPOINT/$TARGET

if [ -e $SOURCE_MOUNTPOINT/.snapshots/$TARGET/last ];
then

    echo "Send incremental snapshot"
    /usr/bin/btrfs send -vvv -p $SOURCE_MOUNTPOINT/.snapshots/$TARGET/last $SOURCE_MOUNTPOINT/.snapshots/$TARGET/$TARGET@$TIMESTAMP | /usr/bin/btrfs receive -vvv $DESTINATION_MOUNTPOINT/$TARGET/

else

    echo "Send full snapshot"
    /usr/bin/btrfs send -vvv $SOURCE_MOUNTPOINT/.snapshots/$TARGET/$TARGET@$TIMESTAMP | /usr/bin/btrfs receive -vvv $DESTINATION_MOUNTPOINT/$TARGET/

fi

# create the source 'last' link to reference
unlink $SOURCE_MOUNTPOINT/.snapshots/$TARGET/last
/usr/bin/ln -s $SOURCE_MOUNTPOINT/.snapshots/$TARGET/$TARGET@$TIMESTAMP $SOURCE_MOUNTPOINT/.snapshots/$TARGET/last

# create the destination 'last' link to reference
unlink $DESTINATION_MOUNTPOINT/$TARGET/last
/usr/bin/ln -s $DESTINATION_MOUNTPOINT/$TARGET/$TARGET@$TIMESTAMP $DESTINATION_MOUNTPOINT/$TARGET/last

First probelm:
Now the problem is every time i have a look into the created "last" symlinks the displayed content is not the current one. But "ls -la" is showing that the synmlink points to the last snapshot dir.

My nasty suspicion is that it is wrong here to create symlinks, isn't it?
But I need a static reference to the last taken snapshot to create an incremental one.
Sure I could sort the snapshot names or try to figure it out via the included timestamp but thats not nearly as safe as creating a symlink reference.

Second problem:
Is it safe to delete some of the older taken incremental backup snapshots?
I tried it out and as long as the taken parent snapshot used for creating the incremental one is present the send/receive of the incremental one is successful.

Maybe I get some up to date information here.
Most howtos I have seen are talking about the risky mysterium "btrfs" but suse is using it as the default fs for their enterprise systems ?!?

Offline

#4 2016-02-19 21:14:04

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: Backup Solution that handles incremental and rename/move

Enterprise users don't have fast moving systems. They have one, tried and true LTS version that has been debugged to within an inch of it's life. That's what people pay Red Hat for. In over a year and a half, I haven't had any real data loss. Several scares, but my data did come through. The only reason I am so sure that my data was okay, however, was because I had a back up on an EXT4 partition which I could compare it to. Btrfs has become considerably more stable since I started, and chances are you could use it for your back up. In fact, I have started using it for my external back up (my main backup is an internal HDD on a seperate SATA controller in the same computer. Best practice? No. Convenient? Hell yes.) largely for the data integrity checking (my internal HDD are enterprise grade, my external are whatever is cheapest).

A snapshot is not a symlink. Here is my understanding. The meta data says "there be data" and points to some data. That is the actual file -- the meta data, not the real data. You can have two files pointing to the same data. They are different files. Deleting one deletes the meta data, but not the real data. As long as a file is using the data it is kept. My daily back up script makes snapshots and deletes the oldest snapshot. The deletion of this oldest snapshot does not effect the other snapshots. The newer snapshot does not reference the old one, but rather references the same data instead.

edit: Oh, the symlink you are refering to was in that script! Sorry. Yeah, I've seen that. I looked at it funny then wrote my own.

The relevant btrfs part is

btrfs sub del $RS1/9/*
btrfs sub del $ARS/9/*
btrfs sub del $JDS/7/*
btrfs sub del $RS0/5/*

for X in {8..1}; do mv $RS1/$X/* $RS1/$(($X+1))/; done
for X in {8..1}; do mv $ARS/$X/* $ARS/$(($X+1))/; done
for X in {6..1}; do mv $JDS/$X/* $JDS/$(($X+1))/; done
for X in {4..1}; do mv $RS0/$X/* $RS0/$(($X+1))/; done

btrfs sub snap $AR/nstgc5/nstgc_local $AR/snapshots/1/nstgc_local.snap_$DATE
btrfs sub snap $AR/nstgc5/nstgc_config $AR/snapshots/1/nstgc_config.snap_$DATE
btrfs sub snap $AR/nstgc5/nstgc_mozilla $AR/snapshots/1/nstgc_mozilla.snap_$DATE
btrfs sub snap $AR/nstgc5/nstgc_local $AR/snapshots/1/nstgc_Tbird.snap_$DATE
btrfs sub snap $AR/arch $AR/snapshots/1/arch.snap_$DATE
btrfs sub snap $R1/nstgc5 $RS1/1/nstgc5.snap_$DATE
btrfs sub snap $R1/pix $RS1/1/pix.snap_$DATE
btrfs sub snap $R1/docs $RS1/1/docs.snap_$DATE
btrfs sub snap $R1/vids $RS1/1/vids.snap_$DATE
btrfs sub snap $R1/music $RS1/1/music.snap_$DATE
btrfs sub snap $JD/dl $JDS/1/dl.snap_$DATE
btrfs sub snap $R0/games/win32/steam $RS0/1/steam-win.snap_$DATE
btrfs sub snap $R0/games/win32/cxoffice $RS0/1/cx.snap_$DATE
btrfs sub snap $R0/games/win32/wineprefixes $RS0/1/wine.snap_$DATE
btrfs sub snap $R0/games/native/steam $RS0/1/steam.snap_$DATE

sync

edit2: Needless to say, Marco's script is far more powerful than anything I've put together. Also note that I'm making writeable snapshots. These can be moved, but they cannot be sent. Marco's uses read only snapshots which cannot be moved, but can be sent. I haven't yet made a decent script for handling read only snapshots since I keep track of their age by means of moving them around.

Last edited by nstgc (2016-02-19 21:41:57)

Offline

Board footer

Powered by FluxBB