Backup of Server Files

LWhitson2 · 2012-02-09 02:01:52

Alright, I have read through the Arch Wiki on Backup options and need some advice. I have a RAID 5 system (~3TB) that I use to store all of mine and my wifes data. I would like to create a weekly backup of this data on an external hard drive (Size TBD, ideally ~1TB). What is the best way to do this?

From my reading, it seems like RSYNC will create a Weekly Backup with Daily Incrementals, but I don't see an option to compress these files. The other option I noticed was Arch-Backup, but I didn't see any documentation on it and there is no information in the Wiki.

With all this being said, what is my best compression option to get a 3:1 compression of my data. For reference, this data is mostly music, pictures, and pdf documents.

Thanks,
Bryce

karol · 2012-02-09 02:19:03

I think you should read a bit about compression.
For text files you want lossless compression, for multimedia you might prefer lossy e.g. instead of wav files, you keep 128 kbit/sec mp3s. You can't compress these mp3s much more using a general-purpose compression program like gzip or xz (2-3% is doable but takes a long time).

It might not be possible to get you a 3:1 compression.
Run a test: pick some files of each type and compress them with different apps, like bzip2, xz, lrzip etc. See how much you can shrink them on different levels of compression and how much time it takes.

If you want, compress each type individually using a dedicated codec: shrink pictures and use a more lossy compression or pick another format e.g. using pngs for scanned images in not always the best idea. Same with music - recode at a lower bitrate or with a lossy codec.

Last edited by karol (2012-02-09 02:25:47)

fukawi2 · 2012-02-09 07:48:06

I'd recommend dar. It supports full and differential backups (and incremental, but I don't like them), compression and encryption. Devote an hour to learning how it works and the syntax etc and works very well.

zenlord · 2012-02-09 17:35:05

I'm using rsync to do my daily server backup, but it is more of a sync really: I can just unmount my external drive and hook it up to another box, use the data there for a while and when I return the box to my server, syncing will resume.

But the compression rate you're looking for will not be easy, unless most of your data is music encoded in WAV and you're willing to compress it to APE, FLAC or even MP3. But if you'll be doing that every week for 3TB of music, than I don't think your server will be doing a lot of other activity then compressing and copying

In other words: I don't think what you're trying to achieve is realistically possible.

Arch Linux

#1 2012-02-09 02:01:52

Backup of Server Files

#2 2012-02-09 02:19:03

Re: Backup of Server Files

#3 2012-02-09 07:48:06

Re: Backup of Server Files

#4 2012-02-09 17:35:05

Re: Backup of Server Files

Board footer