Backup script

wooptoo · 2009-10-24 01:35:29

I created a backup script for my data & system and i'd like to share it.

I have a 320GB hdd (1) as my main hard drive, and a 160GB hdd (2) as a backup drive (i keep this one in a drawer nearby, disconnected from the computer).
What i wanted was to create a partial copy of 1 on 2, such that 2 would contain the entire system (bootable) + a part of my data.

1. You need to have the same filesystem on both (i have ext4). Maybe it works with a mix of ext3 and 4, but it's better to have just one. The partitioning scheme on hdd 1 doesn't matter, but for hdd 2 you need a single partition to which you backup.

2. The scripts:

backup

#!/bin/sh
# rsync backup script

sudo rsync -av --delete-excluded --exclude-from="excl" / "$1"

sudo touch "$1"/BACKUP

This one is very simple. You rsync in archive mode (which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved) and exclude the files that match the patterns from excl.
The / is the source from where you backup (in our case the whole root) and "$1" is the destination to where you backup (this is passed as an argument to the script).

excl

# Include

+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero

+ /media/win
+ /var/run/mpd

+ /home/wooptoo/music/Amestecate
+ /home/wooptoo/music/script
+ /home/wooptoo/music/list.txt
+ /home/wooptoo/music/.hg*


# Exclude

- /home/wooptoo/dl/*
- /home/wooptoo/games/kits/*
- /home/wooptoo/mov/*
- /home/wooptoo/music/*

- /dev/*
- /media/*
- /media/win/*
- /mnt/*
- /proc/*
- /sys/*
- /tmp/*
- /var/run/*
- /var/run/mpd/*

This is a bit more tricky. It's an exclude (and include) file in rsync format.
Exclude: I excluded my games, movies and music from the backup, and also the system directories /dev, /media, /mnt, /proc, /sys, /tmp, /var/run. These are excluded because their content is created at runtime by the system. Note that the direcotries themselves are preserved (you need them!) but they are empty.
Include: even though i excluded /dev, i need to include 4 file from it (which are not dinamically created by udev), these are console, initctl, null, zero.
I also included the directories /media/win and /var/run/mpd. But these are empty, because their content was excluded (in the exclude section).

3. So we got these two files: backup and excl.
Mount the backup hdd, let's say at /media/backup/ and run the script:

./backup /media/backup/

rsync will backup the whole root to that destination. I excluded game kits, music and movies from my backup because they are just too large to fit on hdd 2, and it would also take a lot of time to backup and keep in sync afterwards.

4. After the sync is finished you need to install a boot loader on hdd2, so you can have a working copy of your system.
Open the grub console and type in:

root (hd1,0)
setup (hd1)

The root command tells grub where your system is installed (in this case hdd 2, first partition).
Setup tells grub where to install the boot loader. In my example it is installed in the MBR of hdd 2.

The problem here is that the boot loader installs correctly, but its menu entries are for the partitions of the main system, not the backup system. So if you'll try to boot the backup system, it won't work.
You can fix this by creating a custom menu.lst for the backup hdd. But i prefer not to do this, in order to have an accurate copy of my data. I just prefer to edit the entries from the boot menu on the fly if i need to boot the backup directly. But you can automatically add a custom menu.lst to the backup hdd from the backup script:

#!/bin/sh
# rsync backup script

sudo rsync -av --delete-excluded --exclude-from="excl" / "$1"

sudo cp ~/custom.menu.lst "$1"/boot/grub/menu.lst

sudo touch "$1"/BACKUP

5. Reboot and try out your new system.
I think this approach (system + data backup) is better than just data backup because if something goes wrong with the main hdd, you can always swap in the backup one and continue working. Besides this, you now have another working system, from which you can recover the main one without the need of live CDs.

In my setup both hdds are SATA with AHCI, so they are hot-pluggable. You can plugin the backup drive, run the backup script, and disconnect it. This is very advantageous because you don't have to reboot.
But you can use an USB stick/hdd as backup drive if you only have IDE.

I would like to know what do you think of my backup strategy. Is it good or am i doing it wrong? Are there better methods? What backup strategy do you use? etc.

Last edited by wooptoo (2009-10-24 01:40:46)

brisbin33 · 2009-10-24 03:04:07

wooptoo wrote:

What backup strategy do you use? etc.

I apologize if my response was not your intent, but that's a bit of a loaded question.

you're system if very nice indeed, the ability to swap a drive and keep right on working is great. personally i have a backup script that's cronned to run daily to /mnt/data/backup_daily and monthy to /mnt/data/backup_monthly. it does the following:

makes archive copies of /home, /etc, /usr, /var, and /boot. makes a list of installed pacman packages and a list of installed aur packages.

i then have a restore script which is meant to be run immediately following a catastrophe and subsequent reinstall that does the following:

restores /boot, /home, and /var, reinstall pacman packages from /var via pacman list, restore /etc and /usr, resinstall aur packages from ~/Packages via the aur list. it also takes care to use the proper fstab and menu.lst files.

a side beneifit of this that i didn't realize until recently is that i have a day old copy and a month old copy of virtually all the interesting files on my system. thankfully, i've never had to use ~/.bin/restore, but on a daily basis i do get to use ~/.bin/retrieve:

//blue/0/~ retrieve .xmonad/xmonad.hs 
/home/patrick/.xmonad/xmonad.hs exists, overwrite? [Y/n] y
`/mnt/data/backup_daily/patrick/.xmonad/xmonad.hs' -> `/home/patrick/.xmonad/xmonad.hs'
//blue/0/~ retrieve -m Documents/wifi.txt
/home/patrick/Documents/wifi.txt exists, overwrite? [Y/n] y
`/mnt/data/backup_monthly/patrick/Documents/wifi.txt' -> `/home/patrick/Documents/wifi.txt'

it's so useful to say, shit i just effed up my bash prompt... i'll just restore bashrc from yesterday and start over...

fsckd · 2009-10-24 03:43:08

What hardware do you guys use? Just curious.

dmz · 2009-10-24 06:06:43

@ brisbin, you forgot to share your scripts (your httpd isn't working atm).

Gen2ly · 2009-10-24 08:38:12

wooptoo, this is good. I've been thinking about using rsync for a full system backup. Glad someone figured out /proc, /sys... A question. Why do you use --delete-excluded? Hmm. I've always used --delete to remove files that are on the source but not the backup.

--delete                delete extraneous files from dest dirs
--delete-excluded       also delete excluded files from dest dirs

Could you, explain the difference? I also use '--force' to complete '--delete' (force deletion of dirs even if not empty).

Edit: Ok, I think I get it. '--delete-excluded', removes files that are on backup that are in the exclude list. Right?

wooptoo · 2009-10-24 11:08:57

@Gen2ly: yep, '--delete' wasn't enough because it would leave extraneous files in the excluded dirs.
@fsckd: no special hardware, you can basically do this with any computer.

Last edited by wooptoo (2009-10-24 11:14:25)

brisbin33 · 2009-10-24 23:15:36

dmz wrote:

@ brisbin, you forgot to share your scripts (your httpd isn't working atm).

*EDIT* thank you for pointing this out! i had moved all my pages to a /pages/ subdir of my site. hopefully this sig was the only thing i forgot to update . all should work now. scripts are backup, restore, and retrieve.

Last edited by brisbin33 (2009-10-24 23:18:36)

bunjacn · 2011-05-06 19:31:01

Just to say............ (i hope that this isn't against forum rules):
I ran the script "./backup.sh /media/Podaci" cause i'm backing up to my external HDD. When i ran it, it started deleting the files on the drive, without warning or anything. Fortunately those deleted files weren't important, but you should set the script to ask the user if he wants to do such thing

demian · 2011-05-06 19:41:38

^ made my day
Maybe read the script first? ^^

bunjacn · 2011-05-06 19:55:09

How should i write it then............ there's a warning on the wiki "Warning: don't forget to also exclude the mounted directory where you'll put the backup to avoid an infinite loop (in this example /media/backup/)." I excluded it and it started deleting the files...

Ramses de Norre · 2011-05-06 20:05:45

I use the following to rotate four backups and backup my root and home separately and hard-link common files between all the backups. I run this weekly to have backups of the last four weeks while using barely more space then one full backup would. It is not super customisable as it is my personal script and I've been wanting to add in a check to see whether the backup location is actually mounted, but I've not gotten to it just yet

#!/bin/bash

# Script to rotate three backups of / and /home and make a new incremental
# backup. No arguments required. Make sure the correct disk is mounted at
# $PAR, though!
#
# Original command ran to create the first backup of /home :
# /usr/bin/sudo /usr/bin/rsync --progress --stats -avz \
#     --exclude-from=/home/ramses/home_backup_excludes --delete-exludes \
#     /home/ /media/seadisc/home_backup.0

# Variables and paths
PAR="/media/seadisc/backup"
HOME_EXCLUDES="/usr/local/bin/backup_config/home_backup_excludes"
ROOT_EXCLUDES="/usr/local/bin/backup_config/root_backup_excludes"

SUDO="/usr/bin/sudo"
MV="/bin/mv"
RM="/bin/rm"
RSYNC="/usr/bin/rsync"
DATE="/bin/date"
TEE="/usr/bin/tee"

# Home backups
echo "Moving previous /home backups ..."
$SUDO $RM -rf $PAR/home_backup.3

$SUDO $MV $PAR/home_backup.2 $PAR/home_backup.3
$SUDO $MV $PAR/home_backup2_date $PAR/home_backup3_date

$SUDO $MV $PAR/home_backup.1 $PAR/home_backup.2
$SUDO $MV $PAR/home_backup1_date $PAR/home_backup2_date

$SUDO $MV $PAR/home_backup.0 $PAR/home_backup.1
$SUDO $MV $PAR/home_backup0_date $PAR/home_backup1_date

echo "Doing incremental backup of /home ..."
$SUDO $RSYNC --progress --stats -av \
    --exclude-from=${HOME_EXCLUDES} \
    --delete --delete-excluded \
    --link-dest=$PAR/home_backup.1 \
    /home/ $PAR/home_backup.0
$SUDO $DATE | $TEE $PAR/home_backup0_date > /dev/null

# Root backups
echo "Moving previous / backups ..."
$SUDO $RM -rf $PAR/root_backup.3

$SUDO $MV $PAR/root_backup.2 $PAR/root_backup.3
$SUDO $MV $PAR/root_backup2_date $PAR/root_backup3_date

$SUDO $MV $PAR/root_backup.1 $PAR/root_backup.2
$SUDO $MV $PAR/root_backup1_date $PAR/root_backup2_date

$SUDO $MV $PAR/root_backup.0 $PAR/root_backup.1
$SUDO $MV $PAR/root_backup0_date $PAR/root_backup1_date

echo "Doing incremental backup of / ..."
$SUDO $RSYNC --progress --stats -av \
    --exclude-from=${ROOT_EXCLUDES} \
    --delete --delete-excluded \
    --link-dest=$PAR/root_backup.1 \
    / $PAR/root_backup.0
$SUDO $DATE | $TEE $PAR/root_backup0_date > /dev/null

fukawi2 · 2011-05-07 00:47:33

wooptoo wrote:

What backup strategy do you use? etc.

rdiff-backup to push backups to remote server/location.
rsnapshot to pull backups to a central location.

phildg · 2011-05-07 09:14:04

I've put a lot of thought into my whole storage solution, heres how it works:

My desktop has 4 drives in RAID10, RAID1 would be just as suitable. So I can (and have done) tolerate failure of a drive without downtime. I don't backup system files, just data, config, and downloaded packages, in case I need to reinstall.

My old desktop has 2 drives in RAID1 and sits in a corner out of the way.

My backup runs from cron, and uses WOL to start the old desktop up, iscsi to connect to the storage on that machine, then backup up to it, and finally shut it back down. I get an email of the results.

The actual backup itself uses rsync and hardlinks and works in a similar way to Time Machine, recent back ups are more often, older ones are more spread out. Using hard links means each backup represents a snapshot of what the filesystem looked like at the time the backup was done, rather than having a base and incrementals or deltas.

fukawi2 · 2011-05-08 01:04:52

phildg wrote:

...and finally shut it back down.

How do you do that out of curiosity? Just keyless SSH?

phildg · 2011-05-08 04:44:01

fukawi2 wrote:

How do you do that out of curiosity? Just keyless SSH?

I think you mean unencrypted private key, in which case yes.

Though the backup machine I've restricted what that key is allowed to be used for, and that's just to run /sbin/shutdown -h now

fukawi2 · 2011-05-08 09:46:47

phildg wrote:

I think you mean unencrypted private key, in which case yes.

Uhhh, yes. lol. Sunday mornings I'm allowed to be stupid
Thx for the info.

Convergence · 2012-04-22 22:56:34

Edit:
I should have read more carefully. Nevermind.

Last edited by Convergence (2012-04-22 23:01:45)

ngoonee · 2012-04-26 06:41:00

2009 post. Closing.

Subject is more suitable for a wiki anyway. And there ARE wiki pages dedicated to backup in any case, all round the net.

Arch Linux

#1 2009-10-24 01:35:29

Backup script

#2 2009-10-24 03:04:07

Re: Backup script

#3 2009-10-24 03:43:08

Re: Backup script

#4 2009-10-24 06:06:43

Re: Backup script

#5 2009-10-24 08:38:12

Re: Backup script

#6 2009-10-24 11:08:57

Re: Backup script

#7 2009-10-24 23:15:36

Re: Backup script

#8 2011-05-06 19:31:01

Re: Backup script

#9 2011-05-06 19:41:38

Re: Backup script

#10 2011-05-06 19:55:09

Re: Backup script

#11 2011-05-06 20:05:45

Re: Backup script

#12 2011-05-07 00:47:33

Re: Backup script

#13 2011-05-07 09:14:04

Re: Backup script

#14 2011-05-08 01:04:52

Re: Backup script

#15 2011-05-08 04:44:01

Re: Backup script

#16 2011-05-08 09:46:47

Re: Backup script

#17 2012-04-22 22:56:34

Re: Backup script

#18 2012-04-26 06:41:00

Re: Backup script

Board footer