from-scratch minimalist backup implementation?

ParanoidAndroid · 2014-01-25 02:15:22

I like the script, but again: ideally, I'd like one archive.

I looked at rdiffdir, and I like it. I'm thinking:

1. make a full signature with rdiffdir of /
2. pipe that to bzip2
3. encrypt and pipe to tar
4. all later backups: extract or read orig.sig through pipe to rdiffdir delta, then compress&encrypt the resulting file and append it to the original tar archive containing the signature file.

As I understand it, this is more or less how Duplicity works but with everything bundled up at the end. I could also copy my entire filesystem into a folder inside the tarball, then generate the sig and deltas. That's exactly what the above script does, but packaged into a single archive at the end. Which is what I'm aiming for.

the problem is rdiffdir. It's part of duplicity, and the only other implementation of a directory-capable version of rdiff is a script that leverages rdiff. This would work, except it requires a ton of disk space to function. This is unacceptable in my case.

Is there another program that does this? That is, another program that can generate sig and delta files? Could this be done with rsync alone, somehow? I ask because I'd like to be able to restore from a backup with utilities that are already installed on a default arch or other Linux system.

Even if I'm willing to settle with keeping duplicity around for the sake of having rdiffdir, I've run into a problem. I'm trying to

rdiffdir sig ~/test - | lrzip -b | tar -cf test.bak -

All the steps work fine except for the bit with tar. Now, I read the man page and discovered that tar requires filenames to create archives. It can't read from stdin.

Normally I'd just write the output of the first two to a tmpfile and then run the tar end of things on that. However, since I'm backing up my whole system and I'm running off of an SSD I'd like to avoid disk writes like that. Is there a way to do so with this method?

EDIT:

I found a small program called rdup, which can be used to generate full backups and incrementals, and relies on external programs to handle the compression, encryption, etc.

I think this is what I've been looking for. I'll run some experiments and post my solution, if I find one.

----

Turns out it isn't. Or at least, I can't get it to do what I need. I've asked it to find the files which have changed and then update the tar archive with those files. Still can't figure out how to factor encryption into this.

I'll give hdup a go. It looks more usable.

----

Also not quite it.

So far, of all of the programs I've tried, duplicity seems the best. However, being unable to restore my system with basic commands such as gpg, tar, and bzip2 is a major barrier for me.

Can anyone suggest a backup system that:
- allows encryption
- allows compression
- produces a single, updateable archive, or a series of archives - one original/snapshot, and the date-stamped incremental diffs.

----

btar seems good. It has the features I need, and I believe that I can get similar delta behavior to what I had with rsync along with the encryption/compression and one-file backup I've been trying for... if I'm understanding btar -h correctly.

Last edited by ParanoidAndroid (2014-01-25 05:44:34)

firecat53 · 2014-01-25 06:05:49

Restoring duplicity by hand.

Scott

ParanoidAndroid · 2014-01-25 23:37:51

That still requires rdiff.

I've been experimenting with btar, and it will create exactly the result I'm looking for... except it's proving impossible to unpack with or without btar. The net result of any attempt is something about a broken pipe. It's proving too difficult to be practical.

I'd like to find a solution such as the one I've been describing, but if nothing is available or the concept I've proposed is otherwise impossible, I'll just rewrite my old backup script in BASH and use the hardlinked snapshot idea inside encrypted backup disks.

On that note, that hardlinking idea is great... but would it be more space-efficient to generate rsync batch delta files against the original snapshot, compress them into timestamped bzip2 archives, and then store them on the same disk as the original snapshot folder?

Last edited by ParanoidAndroid (2014-01-25 23:50:09)

firecat53 · 2014-01-26 02:02:02

You do realize that rdiff is provided by librsync, which has no other dependencies other than popt, zlib and bzip2? No python required, if that's what you're concerned about. In addition, all the duplicity/rdiff/rdiff-backup solutions are officially supported in the Arch repos. Btar is AUR or install yourself.

At some point, with most things remotely complex, you have to draw a line and accept a certain amount of dependencies. I understand minimalism....I run monsterwm and a bunch of console programs for the most part! Unless you're truly in it just for the learning experience, it seems like you're trying to scratch a very particular itch...when modifying the nature of your itch just slightly would put the solution well within the bounds of existing tools I admire your thought process on this...but just how much extra work do you really want to put in here?

Scott

Edit: sorry...I hope it doesn't sound like I'm trying to shoehorn you in to _my_ solution...because I'm not! There are many ways to skin the proverbial cat

Last edited by firecat53 (2014-01-26 02:04:35)

ParanoidAndroid · 2014-01-26 02:25:41

Not at all. Your point is well taken, and it's something I've been contemplating throughout this whole endeavor.

I'd like my plan to work, obviously. It would be convenient to have everything wrapped up in a nice, neat bundle. But since my new plan clearly won't work no matter what angle I try, I think I'll re-implement my old one.

Since I can't get rdiffdir without duplicity and I don't like having non-essential stuff on my system, I'm reimplementing my python backup script in a much simpler structure written in BASH. I'll post it when I'm done. Maybe someone will profit by it, and I could use a review of my sloppy BASH coding.

EDIT:

my script is complete, but I can't seem to paste it into the form. Every time I try, uzbl crashes.

EDIT:

I'm running my script through the programming equivalent of WeightWatchers at the moment. The first draft was so bloated and error-prone that it wasn't even funny.

I've got it down to two potential modes of operation. The basic layout is:

1. for each disk in a list of backup disks,
1a. unlock and mount
1b. test to see whether there is a snapshot for yesterday's date.
1ba. if so, then create a hardlinked snapshot for today based on yesterday's. Else, create a full backup for today's date.
1c. unmount and re-lock.

The problem I have now is whether to create snapshots on a day-to-day or a second-to-second basis. If the former, any attempt to make an nth snapshot in the same day merely updates the snapshot for today's date. I'm not sure how to do this without mangling the hardlinks. The latter, simpler solution is to create a timestamp that uses both today's date AND time, which would remove the need to update that day's snapshot. The trade-off is that this would mean a ton more folders to sort through later on and it would make for a bigger size on disk. The benefit is that it would allow recovery of individual files deleted (in theory) say, minutes before.

Last edited by ParanoidAndroid (2014-01-27 23:04:24)

ParanoidAndroid · 2014-01-28 22:03:38

The script is complete, but I can't upload it. At least, I can't copy and paste it. uzbl crashes every time I try. Is there some way to attach it to this post? I'd upload it to a personal website like every other developer I've ever seen, but I'm not really much of a 'developer' and I have no website So. There's that.

I wound up naming the script SPARTACUS - the Slightly PARanoid minimalisT bACUp Script. I also wrote a short script that automates the process of creating backup drives and a sample bashrc file that contains the default environment variables used to configure things like which disks to use, which folders to back up, keyfile, etc.

ewaller · 2014-01-28 22:23:19

ParanoidAndroid wrote:

The script is complete, but I can't upload it. At least, I can't copy and paste it. uzbl crashes every time I try. Is there some way to attach it to this post?

Take a look at :

community/wgetpaste 2.23-1
    A script that automates pasting to a number of pastebin services

pipe your output to it, and you are golden

orschiro · 2014-01-28 22:43:17

Or even better, publish it in the form of a public gist on which everyone can contribute and comment.

https://gist.github.com/

ParanoidAndroid · 2014-01-29 04:20:15

orschiro wrote:

Or even better, publish it in the form of a public gist on which everyone can contribute and comment.
https://gist.github.com/

ewaller wrote:

Take a look at :
community/wgetpaste 2.23-1
A script that automates pasting to a number of pastebin services
pipe your output to it, and you are golden

This. This is fantastic.

I discovered that copying and pasting in uzbl works with the right-click > copy/paste function, but not with the xorg keybindings Ctrl-Insert and Shift-Insert. Odd, but oh well.

I really hate using a rodent.

Here is the link:
https://gist.github.com/ParanoidAndroid77/8681990

I created it as an anonymous paste before I realized that it would be better to create an account... and I was unable to edit the several spelling errors, etc. that I found a few minutes later. So I made an account and then forked it.

Have a crack at it, and I'll check back tomorrow. Goodnight all

Last edited by ParanoidAndroid (2014-01-29 05:24:03)

Arch Linux

#26 2014-01-25 02:15:22

Re: from-scratch minimalist backup implementation?

#27 2014-01-25 06:05:49

Re: from-scratch minimalist backup implementation?

#28 2014-01-25 23:37:51

Re: from-scratch minimalist backup implementation?

#29 2014-01-26 02:02:02

Re: from-scratch minimalist backup implementation?

#30 2014-01-26 02:25:41

Re: from-scratch minimalist backup implementation?

#31 2014-01-28 22:03:38

Re: from-scratch minimalist backup implementation?

#32 2014-01-28 22:23:19

Re: from-scratch minimalist backup implementation?

#33 2014-01-28 22:43:17

Re: from-scratch minimalist backup implementation?

#34 2014-01-29 04:20:15

Re: from-scratch minimalist backup implementation?

Board footer