Arch Linux

skgucek

Member

Registered: 2015-06-22

Posts: 2

My attempt to fix scripts in wiki article Full System Backup with tar

Hello,
recently I installed Arch ARM on my old Raspberry Pi, took a while to set it up as the Pi is slow as hell compared to modern computers. I also take care of a really small VPS for a family business website running debian. Both only have about 16 GB of space, so I was looking into a simple backup solution that would generate compressed tar files that I could later automatically download with ftp to my main machine that has lots of storage space.

I found a wiki article https://wiki.archlinux.org/index.php/Fu … p_with_tar, but found the solution lacking:
a) Why would you need a live cd chroot to perform backups, when you can just tar a live system while excluding system directories and files like shown in https://wiki.archlinux.org/index.php/fu … with_rsync?
b) Can't transparently set compression (might be important for some!)
c) No progress bar (backups can take a long time, which I think warrants pv installation if you run them manually)

I've never really scripted before in bash, so this is my first attempt to fix the aforementioned script. It works great for me (as far as creating an archive goes), but I would like some comments to see if it's done properly. Also, I know that there is a lot of room for improvement, I could envisage a configuration file to avoid editing the script itself to change options, command line switches, etc. But right now I have to study and maybe I will improve it in July, if comments are favourable.

backup.sh

#!/bin/bash
# tar full filesystem backup script with compression

## SET CONFIGURATION HERE
# folder that backups go to
destination=/opt/backup
# path to the exclude file
excludefile=/opt/backup/exclude.txt
# your desired compression command
compression="gzip -9"
# the extension your selected compression format uses
extension="gz"
## END OF CONFIGURATION

# check if script is running as root
if [[ $EUID -ne 0 ]]; then echo "script must be run as root"; exit 1; fi

# check if required files exist
[ -d $destination ] || { echo >&2 "destination folder $destination does not exist"; exit 1; }
[ -f $excludefile ] && exclude="-X $excludefile" || { echo "warning: exclude file $excludefile does not exist, continuing anyway"; }

# check if all required commands are avaliable (might be an overkill but just to be safe)
command -v awk >/dev/null 2>&1 || { echo >&2 "awk is required but not installed"; exit 1; }
command -v du >/dev/null 2>&1 || { echo >&2 "du is required but not installed"; exit 1; }
command -v uname >/dev/null 2>&1 || { echo >&2 "uname is required but not installed"; exit 1; }
command -v date >/dev/null 2>&1 || { echo >&2 "date is required but not installed"; exit 1; }
command -v pv >/dev/null 2>&1 || { echo >&2 "pv is required but not installed"; exit 1; }
command -v mv >/dev/null 2>&1 || { echo >&2 "mv is required but not installed"; exit 1; }
command -v echo "" | $compression >/dev/null 2>&1 || { echo >&2 "$(echo $compression | awk '{print $1}') is either not installed or refusing to read stdin"; exit 1; }

# get size of files and generate a temporary file name for the backup
echo "calculating size, might take some time"
startsize="$(du -sb $exclude --apparent-size / | awk '{print $1}')"
temporaryname="$destination/$(uname -n)-$(date +'%Y%m%d')-temp.tar.$extension"

# the main backup command
{ tar -cPpSf - $exclude --acls --selinux --xattrs / | pv -s $startsize | $compression > $temporaryname; } || { echo >&2 "something went wrong, did not finish"; exit 1; }

# rename the temporary file and get final size
finalname="$destination/$(uname -n)-$(date +'%Y%m%d-%H%M%S').tar.$extension"
[ -f $temporaryname ] && mv $temporaryname $finalname || { echo >&2 "something went wrong, did not finish"; exit 1; }
du -sh $finalname

exclude.txt

#system directories we don't need
/dev/*
/proc/*
/sys/*
/tmp/*
/run/*
/mnt/*
/lost+found

#comment out if you want to backup home directories
/home/*

#if you have a swapfile
#/swapfile

#compressed tar files in your backup directory
/opt/backup/*.tar*

#package manager cache
/var/cache/pacman/pkg/*                 #arch
#/var/cache/apt/archives/*              #debian

Thanks for your comments, maybe this will be interesting or help someone, otherwise see you in July after I'm done with exams! :-)

Last edited by skgucek (2017-06-15 21:09:27)

oliver

Member

Registered: 2007-12-12

Posts: 448

Re: My attempt to fix scripts in wiki article Full System Backup with tar

nice - seeing this reminded me that I need to run a full system backup on some servers at work.

One small point - very last line... useless use of 'echo'?  "du -sh $filename" will give the same results

vacant

Member

From: downstairs

Registered: 2004-11-05

Posts: 816

Re: My attempt to fix scripts in wiki article Full System Backup with tar

Have you looked at Duplicity for easy full and incremental backups? It uses encrypted and compressed tar format.

I also use Rclone to copy my Duplicity archives to Google Drive.

skgucek

Member

Registered: 2015-06-22

Posts: 2

Re: My attempt to fix scripts in wiki article Full System Backup with tar

nice - seeing this reminded me that I need to run a full system backup on some servers at work.

One small point - very last line... useless use of 'echo'?  "du -sh $filename" will give the same results

Thanks for your comment, I edited the script and got rid of it. As I said, this is my first "serious" bash script ever and of course I left in stupid stuff. I think this echo was a leftover from some previous variant that printed something else at the end.

Have you looked at Duplicity for easy full and incremental backups? It uses encrypted and compressed tar format.

I also use Rclone to copy my Duplicity archives to Google Drive.

Unfortunately, I didn't. I was searching for "arch backup" on Google and stumbled upon the aforementioned article https://wiki.archlinux.org/index.php/Fu … p_with_tar. Found it lacking but didn't search further, in true RPi sprit I tried to improve the solution! :-)

Anyways, if anybody runs this on a live machine, especially on servers, I recommend running it as

# nice -n19 ionice -c2 -n7 sudo ./backup.sh

This way CPU utilization and IO operations will be put at the lowest priority possible so it won't throttle your server/machine ... it will take longer to finish but shouldn't drastically reduce responsiveness of the machine. Of course don't do anything dramatic while it's running as it might produce an inconsistent backup.

Last edited by skgucek (2017-06-15 21:21:00)

pyro@Linux

Member

Registered: 2017-11-23

Posts: 1

Re: My attempt to fix scripts in wiki article Full System Backup with tar

I also tried to follow the wiki article, Full system backup with tar, and found the write-up/solution a bit annoying since I want to be able to just run a back-up on my current running system. Your solution above offers an easy way to create a back-up whenever I feel the need without a whole package of overhead. You should create a script to restore from the file created. Maybe I will take a crack at it myself.

loqs

Member

Registered: 2014-03-06

Posts: 17,321

Re: My attempt to fix scripts in wiki article Full System Backup with tar

You should also be aware of https://bbs.archlinux.org/viewtopic.php?id=146269 when using tar instead of bsdtar for backup of executables.