[bash]Rotorsync: A restore-points type snapshot script using rsync.

Thme · 2013-01-02 06:21:01

First a clarification: This script is not a replacement or alternative to existing backup solutions and other frontends for rsync. I mainly wrote it on my netbook to conserve resources and simplify the options I may want to change later on. The result was Rotorsync. It is not a full featured backup tool. Instead it is designed to make one simple purpose a relatively simpler task to anyone inexperienced with rsync and/or unfamiliar incremental snapshots and the uses of hard-linking to conserve disk usage. The purpose is to keep and run backups that are schedule based on a set number of days, and rotate/remove old backups that excede a set number. Both numbers along with the path to where the snapshots will be kept are settings the user defines in /etc/rotorsync.conf
There are 4 required settings: Username, BACKUPTO, HOWMANY & HOWOFTEN. And some Optional settings.

Username= is exactly what it means. set it to the main user you wish to have configs backed up for, In other words: Yourself.

BACKUPTO= this sets the path for where the backup will be stored
this accepts either a partition through /dev/sdXy or /dev/disk/by-uuid/
in the case of the partition it will create its backup folders on the root of that partition by default. If a subdirectory is wanted then edit the SUBDIR= optional setting. This assumes the full path to the mountpoint of the partition so only the subdirectory path is needed. "BACKUPTO=/home/myusername/snapshots" will put the snapshot in my home directory under the folder named "snapshots". "BACKUPTO=/dev/sda5" and "SUBDIR=restorepoints" means that if /dev/sda5 is not mounted it will be mounted and the resulting snapshots will be kept in "restorepoints"

HOWMANY= this sets the number of snapshot to be kept. accepts only numbers. "HOWMANY=5" would mean that the 5 most recent snapshots will be maintained and ones exceding that number will be cleaned out.

HOWOFTEN= this sets the period of when shapshots will be taken the number equals a number of days. "HOWOFTEN=7" would mean that incremental snapshots will be taken once a week.
The optional settings are MOUNTOPTS="" which sets mount options instead of the defaults. MOUNTOPTS is mainly offered for other filesystems that need to be mounted differently, keep in mind that to keep fully functional snapshots they need to be on a filesystem that supports the same attributes/permissions features as the one your root "/" resides on. A general rule it to use the same filesystem type as your root.

BOOTDELAY= is really simple. It just sets a sleep value to be run after startup. "BOOTDELAY=3m" would wait for 3 minutes before starting the main processes. "BOOTDELAY=30" would be 30 seconds. It is merely here for convienience (there are other ways to delay it at boot. See "man systemd.timer" If using systemd.

Notes:
It doesn't track time in 24hour format. Out of convienience it'll still run if your previous boot was yesterday at 10pm and you booted today at 7am (only 9 hours technically).
Expect about 20-45 minutes to create the first snapshot after that the average time to create an incremental snapshot for me has been 3-5 minutes and an additional 5 minutes if the script is removing an older backup once the number excedes your settings.
To answer this possible question: Yes it can still run on cron or any variant as long as it's run with root priveleges. just observe what I won't be changing below. other than that It'll still run just fine on cron like any other script.
As for improvements this is still under work. This should work as it is but I'll be improving it as I go. I'll accept certain requests except the following(may change if clarification is needed):

1:This is not and will not be a backup script for personal media or anything that is not part of the system "/" or hidden files in "/home"... There are far less complicated solutions for that.

2:system shapshots on an hour/minute based schedule. Honestly there's no reason for this on a laptop. That can be done with a few lines of bash script and cron if you really feel the need.

Todos:
1: Add full menu driven restore functionality to the script for use on any live medium.(see "Restoring Snapshots" near the bottom of this post for now.)
2: Additional mount handling and checking.
3: Additional backup parameter to sync snapshots to another external medium(see "keeping a duplicate... off site")
4: Allow snapshots for dotfiles of multiple user directories. /home/(jill,jack etc..)/.dotfiles
Current parameters (currently these have to be separated "-f -v" or "-v -f" not "-vf" or "-fv" as They are not part of case statements... that's next though):
-f force rotorsync snapshot to run regardless of schedule.
-v verbose output

#!/bin/bash  
set -e
if [[ -f /etc/rotorsync.conf ]] ; then 
	source /etc/rotorsync.conf
else
	echo '/etc/rotrorsync.conf not found generating one...'
	echo '#RSNAP CONFIGURATION'       > /etc/rotorsync.conf
	echo '#Main settings (required)' >> /etc/rotorsync.conf
	echo 'Username='                 >> /etc/rotorsync.conf
	echo 'BACKUPTO='                 >> /etc/rotorsync.conf
	echo 'HOWMANY='                  >> /etc/rotorsync.conf
	echo 'HOWOFTEN='                 >> /etc/rotorsync.conf
	echo '#Optional Settings'        >> /etc/rotorsync.conf
	echo 'BOOTDELAY='                >> /etc/rotorsync.conf
	echo 'MOUNTOPTS='		 >> /etc/rotorsync.conf
	echo 'SUBDIR='			 >> /etc/rotorsync.conf
	echo 'DONE! Now please edit /etc/rotorsync.conf before continuing!'
	exit 0
fi
if [[ -z "$BACKUPTO" || -z "$Username" ]] || [[ -z "$HOWMANY"  || -z "$HOWOFTEN" ]] ; then
	echo 'You MUST edit /etc/rsnap.conf before running this script'
	exit 1
fi 
ISPATH=$(echo "$BACKUPTO" | cut -d '/' -f 2) 
if [[ -b $BACKUPTO || -h $BACKUPTO ]] && [[ $ISPATH = dev && -z $MOUNTOPTS ]] ; then
	if [[ ! -d /mnt/snapshots ]] ; then mkdir /mnt/snapshots ; fi
	mount "$BACKUPTO" /mnt/snapshots/ & m1=$! 
	MNTPT="$BACKUPTO" && BACKUPTO=/mnt/snapshots && MNTDIR=$BACKUPTO
	if [[ -n $SUBDIR && ! -d /mnt/snashots/$SUBDIR ]] ; then mkdir "/mnt/snaphots/$SUBDIR" ; fi
	if [[ -n $SUBDIR && -d /mnt/snapshots/$SUBDIR ]] ; then BACKUPTO="/mnt/snapshots/$SUBDIR" ; fi
elif [[ -b $BACKUPTO || -h $BACKUPTO ]] && [[ $ISPATH = dev && -n $MOUNTOPTS ]] ; then
	if [[ ! -d /mnt/snapshots ]] ; then mkdir /mnt/snapshots ; fi
	mount "$MOUNTOPTS" "$BACKUPTO" /mnt/snapshots & m1=$!
	MNTPT=$BACKUPTO && BACKUPTO="/mnt/snapshots"
	if [[ -n $SUBDIR && ! -d /mnt/snapshots/$SUBDIR ]] ; then mkdir /mnt/snapshots/$SUBDIR ; fi
	if [[ -n $SUBDIR && -d /mnt/snapshots/$SUBDIR ]] ; then	BACKUPTO="/mnt/snapshots/$SUBDIR" 
	fi
fi &&
if [[ ! -d $BACKUPTO ]] ; then
	mkdir "$BACKUPTO" && cd "$BACKUPTO"
	if [[ ! -d ROOT ]] ; then mkdir ROOT/ ; fi
	if [[ ! -d DOTFILES ]] ; then mkdir DOTFILES/ ; fi
elif [[ -d $BACKUPTO  ]] ; then
	cd "$BACKUPTO" 
	if [[ ! -d ROOT ]] ; then mkdir ROOT/ ; fi
	if [[ ! -d DOTFILES ]] ; then mkdir DOTFILES/ ; fi
fi
cd DOTFILES
if [[ -f .xlist ]] ; then
	xlst=.xlist && cd ..
else
	echo '- .gvfs' > .xlist               
	echo '- .dbus/**' >> .xlist
	echo '- .local/share/Trash/**' >> .xlist 
	echo '- .thumbnails/**' >> .xlist       
	echo '- .cache/**' >> .xlist	          
	echo '- .opera/cache/**' >> .xlist
	xlst=.xlist && cd ..
fi
if [[ $1 =~ -f || $2 =~ -f ]] && [[ -n "$BOOTDELAY" ]] ; then BOOTDELAY=1 ; fi &&
if [[ -n "$BOOTDELAY" ]] ; then sleep "$BOOTDELAY" ; fi
N1="$HOWMANY" ; N2="$HOWOFTEN" ; opts="-aAXh --del"
if [[ $1 =~ -v || $2 =~ -v ]] ; then opts="-aAXhv --del" ; fi
touch --date="$(date -I -d "$N2 days ago") 00:01:00" .old
root=/		dotdirs=/home/"$Username"
if [[ .old -nt .lastrun || $1 =~ -f ]] ; then
	ionice -c3 -p $$ &> /dev/null
	renice +15 -p $$ &> /dev/null
	cd ROOT ; rdirs=( [0-9]*[0-9]/ ) ; rlnk="--link-dest="$PWD/${rdirs[-1]}""
	cd .. && cd DOTFILES ; ddirs=( [0-9]*[0-9]/ ) ; dlnk="--link-dest="$PWD/${ddirs[-1]}"" &&  cd ..
	if (( N1 >= 1 )) ; then cd ROOT 
		if (( N1 > 1)) && [[ -d ${rdirs[-1]} && ${rdirs[-1]} != $(date -I)/ ]] ; then
			rsync $opts --exclude={/dev/*,/proc/*,/run/*,/tmp/*,/media/*,/sys/*,/home/*,/mnt/*} "$rlnk"	$root	$(date -I)/ 
		elif [[ ${rdirs[-1]} != $(date -I)/ && ! -d $(date -I)/ ]] && [[ -d  ${rdirs[-1]} ]] ; then
			mv 	${rdirs[-1]} 	$(date -I)/
			rsync $opts --exclude={/dev/*,/proc/*,/run/*,/tmp/*,/media/*,/sys/*,/home/*,/mnt/*} $root	$(date -I)/
		elif [[ ${rdirs[-1]} = $(date -I)/ || ! -d ${rdirs[0]} ]] ; then
			rsync $opts --exclude={/dev/*,/proc/*,/run/*,/tmp/*,/media/*,/sys/*,/home/*,/mnt/*} $root	$(date -I)/
		fi ; cd .. && cd DOTFILES 
		if (( N1 > 1 )) && [[ -d ${ddirs[-1]} && ${ddirs[-1]} != $(date -I)/ ]] ; then
			rsync $opts --exclude-from=.xlist	"$dlnk"	$dotdirs/.[^.]*		$(date -I)/
		elif [[ ${ddirs[-1]} != $(date -I)/ && ! -d $(date -I)/ ]] && [[ -d ${ddirs[-1]} ]] ; then
			mv 	${ddirs[-1]} 	$(date -I)/ 
			rsync $opts --exclude-from=.xlist 	"$dotdirs"/.[^.]*	$(date -I)/
		elif [[ ${ddirs[-1]} = $(date -I)/ || ! -d ${ddirs[0]} ]] ; then
			rsync $opts --exclude-from=.xlist	"$dotdirs"/.[^.]*		$(date -I)/
		fi ; cd ..
	fi ; unset rdirs && unset ddirs 
	cd ROOT ; rdirs=( [0-9]*[0-9]/ ) ; drct1=$(( ${#rdirs[@]} - N1 )) ; if (( $drct1 <= 0 )) ; then drct1=0 ; fi
	if (( ${#rdirs[@]} > N1 ))  ; then for xdirs1 in "${rdirs[@]:0:$drct1}" ; do rm -rv "$xdirs1" ; done ; fi
	cd .. && cd DOTFILES ; ddirs=( [0-9]*[0-9]/ ) ; drct2=$(( ${#ddirs[@]} - N1 )) ; if (( $drct2 <= 0 )) ; then drct2=0 ; fi
	if (( ${#ddirs[@]} > N1 )) ; then for xdirs2 in "${ddirs[@]:0:$drct2}" ; do rm -rv "$xdirs2" ; done ; fi
	cd .. && touch -d "$(date -I)" .lastrun 
else
	echo 'Rototrsync has already run recently'
fi
if [[ $PWD = /mnt/snapshots && $(findmnt -rno TARGET $BACKUPTO) = $MNTDIR ]] ; then
	cd / ; sleep 3 &&  umount /mnt/snapshots
fi
exit 0

For systemd users simply create a .service file like so and enable it(keep in mind using the BOOTDELAY= setting in /etc/rotorsync.conf:

[Unit]
Description=Rotorsync system snapshot script 

[Service]
Type=oneshot
#The below line can be wherever you put the script.
ExecStart=/usr/bin/rotorsync 
TimeoutSec=0

[Install]
WantedBy=multi-user.target

These steps will be removed as I safely incorporate them into the script when I can get around to it...
Restoring Snapshots from any live medium(heed the trailing slashes!):
Open any terminal and perform the following...

1:mount partitions containing the snapshots, your root and (optionally) /home (note: for safety it is recomended to mount them in separate directories)
Restoring the root tree:

2:rsync -aAX(or -aAXhv if you want readable and verbose output) --del --exclude=/home/* /path/to/restorepoint/ROOT/(date you wish to rollback to)/ /path/to/your/mounted/root-tree/
For restoring dotfiles:
steps 3 and 4 are precautionary

3:mkdir /path/to/mounted/home/username/dotfiles.old

4:mv /path/to/mounted/home/username/.* /path/to/mounted/home/username/dotfiles.old/

5:rsync -aAXhv /path/to/restorepoint/DOTFILES/(date you wish to rollback to)/ /path/to/mounted/home/username/
Additional notes:If fstab has changed since the backups have been made you'll have to edit fstab and regenerate a grub.cfg(possibly reinstall grub) to reflect those changes.(if all goes well you can safely remove "dotfiles.old/"

Keeping A Duplicate of existing snapshots off site (as in a backup hard drive etc.)
1: mount drive and partition w/snapshots.
2: rsync -aAXHhv --del /path/to/snapshots/ /path/to/mounted/drive/somefolder/ (note: the H flag will preserve the hardlinks otherwise you'll wind up with a very large set of full shapshots that are not incremental anymore).
For a general idea of how these backups are made(excluding the use of hardlinks) see the wiki here: https://wiki.archlinux.org/index.php/Fu … with_rsync It was primarily responsible for my inspiration after discovering for myself how effective and reliable rsync could be for this task.
Feel free to comment or offer suggestions/improvements here.

Last edited by Thme (2013-01-02 06:22:33)

JohnWittle · 2013-01-03 21:16:05

I am currently using this to manage my backups. Thankya.

Thme · 2013-01-03 22:33:54

I believe I forgot to mention that it will also accept 1 as a number for HOWMANY and will adapt to updating only one snapshot and renaming it to the date it was updated. Also it will not run more than once in a given day if say you reboot more than once. Expect the restore functionallity soon. Appreciate the feedback. let me know how it works for you and if any errors occur. I'll have an AUR package setup when I get the time to read up on it more. This may likely be my first contribution to the AUR. So enjoy it. my goal is to try and keep it as reliable as possible. As I mentioned I use it on my own netbook and it has saved my @$$ on a few occasions.

Arch Linux

#1 2013-01-02 06:21:01

[bash]Rotorsync: A restore-points type snapshot script using rsync.

#2 2013-01-03 21:16:05

Re: [bash]Rotorsync: A restore-points type snapshot script using rsync.

#3 2013-01-03 22:33:54

Re: [bash]Rotorsync: A restore-points type snapshot script using rsync.

Board footer