You are not logged in.

#1 2008-02-23 20:57:50

ravisghosh
Member
From: Intergalactic Spaces
Registered: 2006-10-12
Posts: 516
Website

Incremental Backup Of Home Desktop-how to

I was looking for some app which can backup specific drives and directories and copy them to DVD and whenever rerun, it must only copy those files which have changed or which have been added newly. Can someone suggest some simple solution since I'm not someone with good technical knowledge.

Last edited by ravisghosh (2008-02-24 09:19:48)

Offline

#2 2008-02-23 22:59:26

vogt
Member
From: Toronto, Canada
Registered: 2006-11-25
Posts: 389

Re: Incremental Backup Of Home Desktop-how to

You probably only need to keep your /home and /etc backed up, since everything else in root can be restored from a list of packages, and your configuration from /etc.

What you want is called an incremental backup.

Just check out the list here.

I'm pretty sure that the arch wiki outlines some solutions too.

Offline

#3 2008-02-24 00:29:09

ravisghosh
Member
From: Intergalactic Spaces
Registered: 2006-10-12
Posts: 516
Website

Re: Incremental Backup Of Home Desktop-how to

I arch wiki, i could not find any detail about backup. However, there are a few apps in repo, like hdup, tar, sbackup. I tried hdup but could not make out how to use its cli interface. Sbackup seems a good option, but it does not directly write in dvd, but rather just creates a dump on hard disk. So, even if it is an incremental backup, one has to copy the whole archive to dvd, effectively no saving of resources by using incremental backup.

Offline

#4 2008-02-24 00:36:58

thayer
Fellow
From: Vancouver, BC
Registered: 2007-05-20
Posts: 1,560
Website

Re: Incremental Backup Of Home Desktop-how to

rsync and a custom bash script sounds like the best solution to me. you could rsync the directories to a temp dir and then burn the dvd of the temp folder.


thayer williams ~ cinderwick.ca

Offline

#5 2008-02-24 02:48:47

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

I use hdup. It's not difficult, and it fits your case. Let me outline how to use it:
1. install it

2. edit /etc/hdup/hdup.conf
here's a (not quite) minimal one:

[global]
# to which dir the archives will be written
archive dir = /vol/backup <- modify this!
# chown the archives to this use
user = <yourusername>   <- modify this!
compression = bzip
compression level = 6
nobackup = .nobackup
# allow restoring to /
force = yes

[my-comp]
# what to backup, separate with ,.
# For directories add closing slash, like /home/
dir = /home/,/var/abs/local/            <- modify this!
# don't include theses directories
exclude = lost\+found/, /proc/, /dev/, /sys/

Comment: the .nobackup thing is the a filename, which if it occurs in a directory, then that directory is not archived. Possible use: 'touch ~/.mozilla/firefox/<somerandomstring>/Cache/.nobackup' creates empty .nobackup file in the firefox cache dir, so that one is not backed up by hdup.

3. as root, run hdup monthly my-comp. This creates a subdir like '2008-02-23' in /vol/backup (or wherever you put that dir) and inside it is a compressed backup.

The 'monthly' parameter means that hdup backups *everything* (which is in the directories you specified in the .conf file)

Next time, you can run hdup weekly my-comp, which now only backups the files that were changed since last 'monthly' backup. If of course creates a new '2008-whatever' directory in /vol/backup, and so you only need to burn this new dir to dvd.

You can go one more step in this scheme, namely hdup daily my-comp, which archives only stuff which differs from last 'weekly' backup.

Once in a while, you should go back to 'monthly' and then you can discard your old backups.

Finally, you can have more things you want to backup, at different times than what's in 'my-comp' scheme. Just create another section in /etc/hdup/hdup.conf:

[my-webpages]
dir = /var/www/

and you're good to go with hdup monthly my-webpages, ....

EDIT: forgot about restoring backups:
1. to restore, you need to have all 'parent' archives (ie. if you want to restore a weekly backup, you also need to have its 'monthly' available)
2. the archives are just .tar.bz2 archives, so to extract something, you need only tar and bzip2 (if something should go horribly wrong)
3. hdup can restore by hdup restore my-comp 2008-02-23 <dir>. It restores things as they've been up to that date specified, and it unpacks the archives to <dir>. To make it simple, just '/' should be fine (use with care!).

Once you are comfortable, there are other goodies, like pushing archives over ssh automatically, or encryption of archives.

Last edited by bender02 (2008-02-24 03:00:54)

Offline

#6 2008-02-24 09:18:34

ravisghosh
Member
From: Intergalactic Spaces
Registered: 2006-10-12
Posts: 516
Website

Re: Incremental Backup Of Home Desktop-how to

Is hdup can be run as root only?

Now, suppose, I run a "monthly" and after a few days "weekly." So, now I have 2 archieved files in /vol/backup. I burn them to dvd. Then after some days I run "weekly" again and burn this file to dvd. So, my dvd has 3 files all together. THen I delete files in /vol/backup to free up space. After this, if I run "weekly" it wont work since "monthly" file is missing. Am I right? In that way, the backed up files will be residing in the hard disk without freeing up space. I it possible that the /vol/backup be on the dvd so that hdup compares files in monthly archive of dvd and create weekly archieve rather than having the files lie on hard disk and so that hdup writes directly to dvd???

Also, when I want to recover certain files, do I need to copy the entire monthly and weekly archives from dvd to /vol/backup or I can just open the archive using xarchive, etc, and extract any particular file.

Offline

#7 2008-02-24 17:15:14

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

1. yes, you have to run it as root; but it can automatically change ownership of archives to a selected user automatically (see hdup.conf)

2. For 'weekly', you need to have the last 'monthly' on the drive. So yes, the simplest possibility is to keep the last 'monthly' and 'weekly' on the hard disk. (Now, I never tried deleting 'monthly' and doing 'weekly' afterwards, but hdup keeps some info in /vol/backup/etc (including filelist), so it might me that it can perform 'weekly' just based on that filelist, without the archive actually present.)
What you can do if you have the archive only on the dvd, is to mount the dvd, and then place a soft link of /media/dvd/2008-whatever to the place where hdup expects to find it: /vol/backup/my-comp/2008-whatever. The command is ln -s /media/dvd/2008-whatever /vol/backup/my-comp/2008-whatever.

3. hdup cannot directly write do dvd. It seems to me that this is the problematic part for most backup programs I've came over. The ones which I've seen have either incremental backups (rdiff-backup, hdup, flyback (gui), rsnapshot, areca (gui), timevault (gui)), or can write to dvd (backup-manager)

4. for recovering, since the backups are just .tar.bz2, you can use any program which knows how to deal with these to extract a file. Any standard archive manager (like xarchive).

Offline

#8 2008-02-24 18:43:11

dw
Member
From: Vienna, Austria
Registered: 2006-11-25
Posts: 160

Re: Incremental Backup Of Home Desktop-how to

i'd probably go with a custom bash script as already suggested as well using duplicity for backup and mkisofs for writing data to dvd.

Last edited by dw (2008-02-24 18:44:28)

Offline

#9 2008-02-24 19:19:11

stonecrest
Member
From: Boulder
Registered: 2005-01-22
Posts: 1,190

Re: Incremental Backup Of Home Desktop-how to

s3sync is awesome.. it uses rsync to backup files to amazon's s3, which is ridiculously cheap.


I am a gated community.

Offline

#10 2008-02-25 10:42:23

ravisghosh
Member
From: Intergalactic Spaces
Registered: 2006-10-12
Posts: 516
Website

Re: Incremental Backup Of Home Desktop-how to

bender, it would be great if you put your post in arch wiki.

Offline

#11 2008-02-25 16:43:18

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

Yes, I was thinking about putting it to wiki already. It's way too long for a forum post smile
So maybe later today...

Offline

#12 2008-02-27 00:43:15

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

Offline

#13 2008-04-16 12:50:55

Cippa Lippa
Member
From: Toronto, ON
Registered: 2007-04-12
Posts: 159

Re: Incremental Backup Of Home Desktop-how to

have you ever tried to restore with hdup? I always get segmentation faults!

Offline

#14 2008-04-16 13:36:13

SpookyET
Member
Registered: 2008-01-27
Posts: 410

Re: Incremental Backup Of Home Desktop-how to

Backing up is not the problem. Restoring is. I've tested a few backup utilities.

rdiff-backup is very nice. I would still use a wrapper script around it. I've found a good one, but it's for locally mounted backups, not ssh. sshfs does not support ACL (not sure about extended attributes). It's transactional; meaning, it's a bitch to backup over wireless. It does not resume a failed backup. It restarts from the beginning. It will not transfer the files again, but it will hash them, which is slow and CPU intensive. It's nice that it supports reverse deltas. There is no full backup then the application of many diffs if you want to restore the latest. There is, however, if you want to restore an older backup. It applies the newest. Then it applies the diffs to get to the older one. So, you can delete all backups older than last.

duplicity: Slow, very slow. Its output is horrible. It has to be parsed by a wrapper script. It creates traditional backups. It does not do reverse diffs. It uses tar and gpg to create encrypted backups. It's still beta.

dar with a wrapper script. It's awesome. The output is good looking and dar stores ACL, extended attributes, and much more. It will restore the files exactly like they were. Unfortunately, the wrapper script does not support restoration. It only creates them. You have to restore manually. This is what I'm using right now. I wish duplicity would use dar instead of tar.

Offline

#15 2008-04-16 14:23:17

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

Cippa Lippa wrote:

have you ever tried to restore with hdup? I always get segmentation faults!

I have, but I restore them manually (it's just tar.gz or .tar.bz2 archive).

By the way, I switched to dar since then, and I like it a lot... except I'm worried about restoring smile since dar uses its own format, so I need to have dar_static around just in case.

EDIT: But as far as ease of setup goes... dar requires some script editing (or writing), so from this point of view I like hdup better. And one thing I miss in dar is gpg encryption... sure, it can encrypt somehow. but...

Last edited by bender02 (2008-04-16 14:56:14)

Offline

#16 2008-04-16 15:16:20

chimeric
Member
From: Munich, Germany
Registered: 2007-10-07
Posts: 254
Website

Re: Incremental Backup Of Home Desktop-how to

bender02 wrote:

And one thing I miss in dar is gpg encryption... sure, it can encrypt somehow. but...

You can run a script/command after each slice has been created automatically using the -E option of dar to encrypt the backups on the fly with gpg.

Dar also supports make like configuration files which just contain the same options you would normally use at the cli, this way you can create different config files for each backup (home/system/whatever) and use them like:

% dar -c full-backup -B configfile

# or for differential backups given you run it in the directory you have the full backup or the dar catalogue

% dar -c diff-backup -A full-backup -B configfile

Offline

#17 2008-04-16 15:26:50

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

chimeric wrote:
bender02 wrote:

And one thing I miss in dar is gpg encryption... sure, it can encrypt somehow. but...

You can run a script/command after each slice has been created automatically using the -E option of dar to encrypt the backups on the fly with gpg.

Thanks, I guess I didn't *really* read the manpage smile

chimeric wrote:

Dar also supports make like configuration files which just contain the same options you would normally use at the cli <snip>

Well, I ended up using Manuel Iglesias' "automatic_backup" script, and I'm (so far) happy with that. But good to know.

Offline

#18 2008-04-16 17:17:23

dschrute
Member
From: NJ, USA
Registered: 2007-04-09
Posts: 183

Re: Incremental Backup Of Home Desktop-how to

... except I'm worried about restoring smile since dar uses its own format, so I need to have dar_static around just in case.

System Rescue CD includes dar :
http://www.sysresccd.org/Main_Page

Also for restoring, dargui is not bad.  Still a little rough around the edges, but it works.

Offline

#19 2008-04-17 01:16:36

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

dschrute wrote:

System Rescue CD includes dar :
http://www.sysresccd.org/Main_Page

Sure, I just like RIPlinux better wink (Also includes dar, just checked.)

Offline

#20 2008-04-17 13:46:16

dschrute
Member
From: NJ, USA
Registered: 2007-04-09
Posts: 183

Re: Incremental Backup Of Home Desktop-how to

Sure, I just like RIPlinux better wink (Also includes dar, just checked.)

Ahh, I didn't realize K. Robotti was still cranking out stuff.  I used his LoopLinux on a very old 386 laptop, way back when it was called something else...Can't remember right now.  You can't go wrong with the stuff he puts together.  Guess I should take a closer look at RIPLinux.

Offline

#21 2008-04-18 07:12:17

rine
Member
From: Germany
Registered: 2008-03-04
Posts: 217

Re: Incremental Backup Of Home Desktop-how to

The script here works fine for me: http://wiki.ubuntuusers.de/Skripte/Backup_mit_RSYNC
It's in german, but you should be able to read the script.
Ziel = target
Quellen = sources

Offline

#22 2008-04-18 14:21:51

SpookyET
Member
Registered: 2008-01-27
Posts: 410

Re: Incremental Backup Of Home Desktop-how to

bender02 wrote:
chimeric wrote:
bender02 wrote:

And one thing I miss in dar is gpg encryption... sure, it can encrypt somehow. but...

You can run a script/command after each slice has been created automatically using the -E option of dar to encrypt the backups on the fly with gpg.

Thanks, I guess I didn't *really* read the manpage smile

chimeric wrote:

Dar also supports make like configuration files which just contain the same options you would normally use at the cli <snip>

Well, I ended up using Manuel Iglesias' "automatic_backup" script, and I'm (so far) happy with that. But good to know.

That's the one I'm using. I wish it had restoration capability.

Offline

#23 2008-04-18 14:30:02

SpookyET
Member
Registered: 2008-01-27
Posts: 410

Re: Incremental Backup Of Home Desktop-how to

chimeric wrote:
bender02 wrote:

And one thing I miss in dar is gpg encryption... sure, it can encrypt somehow. but...

You can run a script/command after each slice has been created automatically using the -E option of dar to encrypt the backups on the fly with gpg.

Dar also supports make like configuration files which just contain the same options you would normally use at the cli, this way you can create different config files for each backup (home/system/whatever) and use them like:

% dar -c full-backup -B configfile

# or for differential backups given you run it in the directory you have the full backup or the dar catalogue

% dar -c diff-backup -A full-backup -B configfile

The problem is that unlike duplicity, you manage restorations manually. It will be a bitch to use gpg, especially if you use dar_manager.

Offline

#24 2008-04-18 14:32:36

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: Incremental Backup Of Home Desktop-how to

rine wrote:

The script here works fine for me: http://wiki.ubuntuusers.de/Skripte/Backup_mit_RSYNC
It's in german, but you should be able to read the script.
Ziel = target
Quellen = sources

I was discussing backups which produce only one archive file (which is easily burned). As far as rsync-type backups go, I liked rdiff-backup (which does more-less the same thing as that script).

Offline

#25 2008-04-19 16:12:10

SpookyET
Member
Registered: 2008-01-27
Posts: 410

Re: Incremental Backup Of Home Desktop-how to

I have added SafeKeep to AUR: http://aur.archlinux.org/packages.php?ID=16463

Last edited by SpookyET (2008-04-19 16:12:23)

Offline

Board footer

Powered by FluxBB