You are not logged in.

#1 2012-11-09 11:57:21

whoops
Member
Registered: 2009-03-19
Posts: 788

[solved] How do I test / finish my backup script?

Hi!

I'm stuck with writing a backup script because I don't know how to test / finish it (and also I suck at bash T_T). Haven't even tried running it once because it'll probably break EVERYTHING forever (... after I eliminated the obligatory stupid syntax errors which I can't find without running the script, that is). Can't just replace the dangerous parts with "echos" for testing, because it's supposed to handle the backups it makes.... so... I need to run it before I can test it I guess? I thought about running it in a vbox, but there wouldn't really be any realistically changing files etc to see if it really works... is there another way to do this?

This script is supposed to:
- Find the one of the right external hds if plugged in (I'm planning to use 2 alternating hds)
- Do a single full system backup
- Keep incremental backups without frequently changing files and unimportant / big files
- Automatically merge folders from old incremental backups in a "smart" way.
( also I'm planning to make the full backup bootable somehow later, but that's another story )

Now after making this draft and going over the lines a hundred times in my head - what next?

#!/bin/bash

# check if root
if [ "$(id -u)" != "0" ]; then echo "Script not running as root."; exit 1; fi

backup_drive="$(ls -d /media/*/BACKUP)"
echo "$backup_drive" | wc -l | grep 1 || { echo "0 or multiple backup drives found. Exactly 1 needs to be attached." ; exit 1; }
# check if identifier file is fine and matches uuid, label, etc
grep "$(blkid $(mount | grep $backup_drive) | sed 's/.*: //')" "$backup_drive/BACKUP/backupdriveisabove" \
	|| { echo "$backup_drive is not a valid backup drive." ; exit 1; }
echo "Backup drive found in $backup_drive"

# Help! I'm scared!
for countdown in `seq 0 9`; do echo "death and destruction in `expr 10 - $countdown`"; sleep 1; done;


# Rsync stuff of which incremental backups should be kept in ./BACKUP/$(date +%Y-%m-%d) first
synctime=1337
while (($synctime < 15)) 
	do statime=$(date +%s);
	rsync -aAXvxbiH /* $backup_drive --backup-dir=$backup_drive/BACKUP/$(date +%Y-%m-%d) --log-file="$backup_drive/BACKUP/$(date +%Y-%m-%d).log" --exclude-from="/opt/userscripts/backup.exclude" --delete --fuzzy
	eval synctime=`expr $(date +%s) - $statime`
	if (($synctime > 14)); do echo "rsync took too long ($synctime) - repeating" 
	done;

# Rsync previously excluded files / nonincremental full backup to backup drive root
rsync -aAXv /* /path/to/backup/folder --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs}

# Try to merge two old backup increments without making too big a gap (if there are more than 60 backups)
minnamecur=0;
treshold=$(ls -d 2???-??-?? | wc -l);
if (($treshold > 60))
	then { 
		treshold=`expr $treshold / 3`
		# Do not create gaps between incremental backups that would be bigger than this many seconds
		mingap=$[60*60*24*30]
		cd $backup_drive/BACKUP || { echo 'incremental backup folder not found' ; exit 1; }
		c=0
		# find backup that would leave the smallest gap
		for dir in 2???-??-??;
			do c="`expr $c + 1`";
			dirsec=$(date -d "$dir 00:00" +%s)
			eval name_$c=$dirsec; 
			if (($c > 4)) && (($mingap > 86399)); 
				then {
					cpre="`expr $c - 4`";
					ccur="`expr $c - 3`";
					cnex="`expr $c - 2`";
					eval namecur=\$name_$ccur;
					eval namepre=\$name_$cpre;
					eval namenex=\$name_$cnex;
					gap=`expr $namenex - $namepre`;
					if (("$gap" < "$mingap"))
						then {
							mingap="$gap"
							export minnamecur="$namecur"
							export minnamenex="$namenex"
							daysgap=`expr $mingap / 86400` 
					} fi;
				# reduce acceptable gap size for newer backup increments depending on amount of backups
				gapreduce="`expr $mingap / $treshold`";
				mingap="`expr $mingap - $gapreduce`";
				# echo "DEBUG: $c -- Folder $namecur -- acceptable gap size reduced to: $daysgap days";
			} fi; 
		done;
		# merge 2 backup increments / remove the old one. 
		if (( $minnamecur > 2000 ))
			then {
				eval rmback=$(date -d "@$minnamecur" +"%Y-%m-%d");
				eval toback=$(date -d "@$minnamenex" +"%Y-%m-%d");
				cp -lrT ./$toback ./$rmback &&\
					rm ./$toback -r &&\
						mv ./$rmback ./$toback &&\ 
				echo moved $rmback to $toback, resulting gap: $daysgap days;
			} else echo "Could not find Folder to delete. Maybe gaps between backups are too big."
		fi;
} fi;

# unmount the backup drive :3
umount $backup_drive -v || { echo "Something went horribly wrong but now it's too late to do anything about it :D" ; exit 1; }

Thanks!

Last edited by whoops (2012-11-23 09:56:38)

Offline

#2 2012-11-09 19:47:18

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 18,114
Website

Re: [solved] How do I test / finish my backup script?

rsync has a --dry-run flag; that's what I'd use to test it...

To get it to handle the files, run it on a couple of dummy directories with sample data.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#3 2012-11-09 20:19:13

whoops
Member
Registered: 2009-03-19
Posts: 788

Re: [solved] How do I test / finish my backup script?

Oh, sorry I didn't write that properly. I have tested the following parts of the scripts while I wrote it:
- I tested the rsync commands (both dry-run and
- Tested the merging part on dummy folders
- I tested the "find-the-smallest-gap"-part on real backups (with moving folders instead of "merging" + deleting) and ran that repeatedly for a day... the resulting gaps looked ok.
- Most of the rest I tested by echo'ing the result of single or multiple lines... and stuff like that.

Then I stitched everything together. And now I'm clueless.

Hmmm... but I can at least eliminate the syntax errors by commenting out all lines that actually do something I guess. Then I can test the commented out lines without the context again... and then... I guess at some point I'll just have to run the whole thing?

Last edited by whoops (2012-11-09 20:25:47)

Offline

#4 2012-11-09 20:24:20

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 18,114
Website

Re: [solved] How do I test / finish my backup script?

whoops wrote:

I guess at some point I'll just have to run the whole thing?

Only after running it on dummy data smile


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5 2012-11-09 20:39:35

whoops
Member
Registered: 2009-03-19
Posts: 788

Re: [solved] How do I test / finish my backup script?

Ok, guess that means I'll write a script that generates better dummy data next....

For now, I commented out the dangerous lines and ran the rest of the script. For some reason, my cdrom closed when I started it (and I almost had a heart attack and hit strg+c)
Tested again, the same thing happened. Maybe I shouldn't be writing scripts like that... because unless I'm mistaken backup scripts have no business closing cdroms.

Need to read blkid manual again now because that seems to be what's doing it T_T

Offline

#6 2012-11-09 21:20:22

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,584

Re: [solved] How do I test / finish my backup script?

Backup is a multi-billion dollar industry.  If you just want to backup your data, use an existing tool.  But if you want to learn to script, you should learn to do it right.  For that, I recommend Greg's wiki.  I didn't find it until a few years after I began scripting, but I still learned more from it than any other single source.  TBH, I'm not sure it will help with any of the issues particular to this script.  I primarily suggest it because there's some things in your script (use of eval, parsing ls) that make me shudder.  But more importantly, there's stuff like

eval name_$c=$dirsec;

that makes me think you could benefit from an understanding of Arrays.

At the very least, the stuff you learn at Greg's wiki will help you do a lot of things more concisely, which will make your scripts much easier to debug.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#7 2012-11-09 21:29:16

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 18,114
Website

Re: [solved] How do I test / finish my backup script?

Seeing I am here: a couple of minor observations:

* parsing ls is a bad idea, you instead use awk or grep to test the output of mount for a reliable way to ensure your backup drive is mounted; similarly with treshold.

* use $(...) instead of backticks

* your date regex looks dodgy smile


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#8 2012-11-16 07:48:11

whoops
Member
Registered: 2009-03-19
Posts: 788

Re: [solved] How do I test / finish my backup script?

Thanks, that wiki looks like it's going to safe me a lot of trouble in the long run! Guess I'll stick with bash for now after all...

I think I used eval, backticks and a workaround instead of an array because I started writing the script on my router over ssh and for some reason $(), arrays and some other things kept failing. Might be the shell  there. But I'm not exactly sure any more why I wanted to run it on my router in the first place... another thing I need to backtrack when I'm done testing that script. I just hope I don't accidentally end up reimplementing the functionality of rsync I'm using in busybox - that could take ages.

Offline

#9 2012-11-20 07:06:56

ernibert
Member
Registered: 2012-03-09
Posts: 6

Re: [solved] How do I test / finish my backup script?

I'd simply use a VirtualBox-VM for such kind of stuff.  You can make snapshots and set the whole machine back to that state when you messed things up.
Keep in mind that backup if good to have, but its probably useless when there is no stable way to restore after a heavy crash (that is, store your restore scripts right near the backups ;-) ).

Offline

#10 2012-11-20 18:07:06

Thme
Member
From: Raleigh NC
Registered: 2012-01-22
Posts: 93

Re: [solved] How do I test / finish my backup script?

A chroot setup may be sufficient enough though for testing some parts of a script. In whoops case here one could manually snapshot his system into a directory with rsync, chroot into that snapshot and use that as the playground for testing and restoring... but IMHO as far as rotating backup are concerned, using the --link-dest option and passing the previous snapshot to it in rsync would likely be more reliable for rotating snapshots of "/". That would ensure nothing is lost between the increments and eliminates needing to merge anything as old data is hardlinked. This way rotation would be as easy as removing the older snapshots after reaching a certain number or date or total size etc... You can safely remove ones in between as well without affecting the others. I did something similar on my netbook and have successfully restored from my snapshots a few times... On an atom n450 and 7200rpm hdd the whole process takes less than five minutes to create a new snapshot and clean out the old ones so it's pretty fast and effecient... Updating daily on a system whose "/" tree is about 10gb with everything installed. I currently have 5 snapshots totaling around 13gb locally on another partition and that also gets backed up to an external drive with rsync using the -H flag to recreate hardlinks between the snapshots. But from what I've read I believe the perl-script rsnapshot can be setup to handle some of the rotating stuff in the manner the OP is trying to achieve. Other suggestions when using rsync in a script that deals with lots of data (ie.. the root "/" tree) are setting the nice and ionice values at the beginning with something similar to these 2 lines. The values can be what you prefer.

ionice -c3 -p $$ &> /dev/null
renice +15 -p $$ &> /dev/null

Also if the script does the mounting and you're running systemd you can write a service file as was suggested to me in this post here https://bbs.archlinux.org/viewtopic.php?id=149419 for detecting the drive. this way It can be mounted by uuid instead providing some ensurance that the right partition gets mounted.
Additional notes when using rsync:
using --link-dest= can also prevent some possible issues regarding the integrity of the original snapshot. For instance. If the parts of the orginal snapshot get accidentally rm -rf'ed or deleted then the increments made are likely to be useless in the event of a serious emergency... --link-dest= isn't perfect as filesystem corruption or corruption on an inode level to the data would render either solution useless.

Last edited by Thme (2012-11-21 13:39:25)


"You are like people in a dark room waiting for someone to turn the light on for you instead of groping around in the dark and turning it on for yourself." -J. Krishnamurti at age 19, to his students-
www.jkrishnamurti.org

Offline

#11 2012-11-23 09:56:18

whoops
Member
Registered: 2009-03-19
Posts: 788

Re: [solved] How do I test / finish my backup script?

Thanks everyone!

Testing went fine so far and the scripts seems to be mostly working. Trying it in Virtualbox was a mess and I haven't been able to get realistic results for the difference between the two backup steps (incremental and nonincremental / full)... But then I replaced most relative paths with absolute ones and tested it in a chroot (system mounted in there read-only) first to make sure it doesn't ruin my real system. Then I used it on the real system on a copy of my backup for a while longer and manually compared the spare backup to the real backup to see if anything important goes missing. Next I guess I have a lot of cleaning up to do... and then testing again... didn't "fix" the array-workaround yet because I still have some hope left I'll figure out how to run this on my router without installing bash there at some point (busybox).

It's a nice feature, but I don't really like using --link-dest in this case because I prefer having only the files that changed inside the old backup increments, not a full snapshot for each one. This backup exists primarily to deal with "human failure", not so much "system failure" and it's a lot easier to detect (rarely used) files I messed up when there aren't so many dupes/hardlinks around. I'm planning to keep at least 2 of those backup drives around and switch them out every now and then... Mounting by UUID instead of searching for the mount sound good. Not sure how I'm going to do this with multiple different backup drives, but I'll look into it.

Also still trying to perfect my exclude lists before I continue modifying the script... There's still a lot of files that leave me uncertain whether I really want incremental backups of them... or just the lastest version in my full backup... or no backup at all. Trying to exclude all files that can be regenerated by the programs that created them from both backups... and the ones that only have unessential but frequent changes from the incremental backup only... firefox and claws-mail are giving me a hard time trying to figure out which files they really need and so do some other programs.

Argh, and also I still need to figure out a way to make the backup drives bootable automatically, so if I ever really ruin my main drive I can just pop one of the backup drives in instead... still don't like the idea of using sed and strange UUID-parsery on the backed up fstab but doesn't look as if I'll get around that if I really want that (rather unimportant but neat) feature.

Script looks like this at the Moment and probably will for a while...:

#!/bin/bash

# check if root
if [ "$(id -u)" != "0" ]; then echo "Script not running as root."; exit 1; fi

# find backup drive marker file
while read mount; 
	do 
		if [ -a "$mount/BACKUP/backupdriveisabove" ]; 
			then echo $mount; 
			backup_drive=$mount; 
		fi; 
	done < <(mount | grep /media | sed "s/.*on \(.*\) type.*/\1/")

# check if marker file is fine - should match: 
# directory="/media/BACKUPMOUNT"; blkid $(mount | grep "$directory") | sed 's/.*: //' > $directory/BACKUP/backupdriveisabove
grep "$(blkid $(mount | grep $backup_drive) | sed 's/.*: //')" "$backup_drive/BACKUP/backupdriveisabove" \
	|| { echo "$backup_drive is not a valid backup drive." ; exit 1; }
echo -e "\E[32m===== Backup drive found in $backup_drive =====" && tput sgr0

# Help! I'm scared!
for countdown in `seq 0 5`; do echo "death and destruction in `expr 5 - $countdown`"; sleep 1; done;

# Rsync stuff of which incremental backups should be kept in ./BACKUP/$(date +%Y-%m-%d) first
echo -e "\E[32m===== Partial incremental backup =====" && tput sgr0
synctime=1337
while (( $synctime > 15 )) 
	do statime=$(date +%s);
		rsync -aAXvxbiH /* $backup_drive --backup-dir=$backup_drive/BACKUP/$(date +%Y-%m-%d) --log-file="$backup_drive/BACKUP/$(date +%Y-%m-%d).log" --exclude-from="/opt/userscripts/backup.exclude" --delete --fuzzy
		eval synctime=`expr $(date +%s) - $statime`
		if (($synctime > 14)); then echo "rsync took too long ($synctime) - repeating"; fi;
	done;

# Rsync (almost all) previously excluded files / nonincremental fully functional backup to backup drive root
echo -e "\E[32m===== Full system backup =====" && tput sgr0
rsync --delete -aAXv /* $backup_drive --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,*/BACKUP/*,/home/*/.ccache/*,/home/*/.thumbnails/*,/var/tmp/*,/home/*/.mozilla/firefox/*.default/Cache/*,/home/*/.mozilla/firefox/*.default/thumbnails/*,/home/*/.pulse/*}


# Try to merge two old backup increments without making too big a gap (if there are more than 60 backups)
echo -e "\E[32m===== Trying to merge 2 backup increments =====" && tput sgr0
minnamecur=0;
treshold=$(ls -d $backup_drive/BACKUP/2???-??-?? | wc -l);
if (($treshold > 60))
	then { 
		treshold=`expr $treshold / 3`
		# Do not create gaps between incremental backups that would be bigger than this many seconds
		mingap=$[60*60*24*30]
		pushd "$backup_drive/BACKUP" || { echo 'incremental backup folder not found' ; exit 1; }
		c=0
		# find backup that would leave the smallest gap
		for dir in 2???-??-??;
			do c="`expr $c + 1`";
			dirsec=$(date -d "$dir 00:00" +%s)
			eval name_$c=$dirsec; 
			if (($c > 4)) && (($mingap > 86399)); 
				then {
					cpre="`expr $c - 4`";
					ccur="`expr $c - 3`";
					cnex="`expr $c - 2`";
					eval namecur=\$name_$ccur;
					eval namepre=\$name_$cpre;
					eval namenex=\$name_$cnex;
					gap=`expr $namenex - $namepre`;
					if (("$gap" < "$mingap"))
						then {
							mingap="$gap"
							export minnamecur="$namecur"
							export minnamenex="$namenex"
							daysgap=`expr $mingap / 86400` 
					} fi;
				# reduce acceptable gap size for newer backup increments depending on amount of backups
				gapreduce="`expr $mingap / $treshold`";
				mingap="`expr $mingap - $gapreduce`";
				# echo "DEBUG: $c -- Folder $namecur -- acceptable gap size reduced to: $daysgap days";
			} fi; 
		done;
		# merge 2 backup increments / remove the old one. 
		if (( $minnamecur > 2000 ))
			then {
				eval rmback=$backup_drive/BACKUP/$(date -d "@$minnamecur" +"%Y-%m-%d");
				eval toback=$backup_drive/BACKUP/$(date -d "@$minnamenex" +"%Y-%m-%d");
				cp -vlrTf $toback $rmback &&\
					rm $toback -rv &&\
						mv -v $rmback $toback &&\
							echo -e "\E[32m===== moved $rmback to $toback, resulting gap: $daysgap days ====="  && tput sgr0
			} else echo "Could not find Folder to delete. Maybe gaps between backups are too big already."
		fi;
} fi;

popd
# unmount the backup drive :3
umount $backup_drive -v \
	|| { echo "Something might have gone horribly wrong but now it's too late to do anything about it :D" ; exit 1; }

This script is far from finished, but I guess this Topic is "solved" because I can do test-runs + try around now that I'm more comfortable with...:
- running the script in general
- mounting my system read-only in a chroot.
- using absolute paths
- UUIDs
... and started reading stuff on this site: http://mywiki.wooledge.org
... and most importantly: have another bootable backup of my spare backup backup. I tried to avoid this, but nothing really beats simply just buying ANOTHER huge spare HD, making sure it works as a full replacement and putting it somewhere far away from the rest of the HD stash (*sigh* although I might drown in HD's one day or suffocate on one that somehow makes its way into my breakfast bowl T_T).

Last edited by whoops (2012-11-23 10:01:06)

Offline

Board footer

Powered by FluxBB