Arch Linux

micman

Member

Registered: 2016-08-10

Posts: 16

Looking for the best way to make the system backup by means of rsync

Hello.

After a fresh installation of Arch Linux–-the last one was two and a half years old and had accumulated several flaws–I'm looking for the best way to make periodic backups thanks to rsync.
The previous method I used was to run Archlinux in text mode during the night, without Internet connection and other partitions mounted separately from the system, executing these commands:

# rsync -vauxHAX --delete --progress / /path/to/backup/folder

Everything worked well for various backups, but the last one I did several months ago failed, and if I remember well, it ended with a warning that the destination partition was full. The partition table of the external hard disk used for the backup has been deleted. Now I have decided to use an entire USB key for periodic backups in order to avoid problems of this type, and, after reading more attentively the wiki instructions (https://wiki.archlinux.org/index.php/Rsync), I thought to change the commands like this:
For the first (complete) backup:

# rsync -aAXHv --progress --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/swap"} / /path/to/backup/folder

For the following (incremental) backups:

# rsync -aAXHv --progress --delete --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt*","/media/*","/lost+found","/swap"} / /path/to/backup/folder

For now, I don't care much about scheduling periodic backups, for example through cron, but rather to make sure that the commands are correct. Then I'll see if it’s better for me to use an automation script or keep doing manual backups every 15 days or a month–for a personal use, I think it’s enough.
If anyone has any good advice to give me, thank you for doing it.
One last question: is there a way to verify that the backup performed by rsync is correct and usable?

drcouzelis

Member

From: Connecticut, USA

Registered: 2009-11-09

Posts: 4,092

Website

Re: Looking for the best way to make the system backup by means of rsync

Oh man, I love talking about backups, and rsync!

So, it looks like the only difference between your "first time" and "incremental backups" command is the "--delete" argument? If that's the case, there's no need to ever use the "first time" command. Both commands will behave the same if the "/path/to/backup/folder" is initially empty.

If you are interested in true incremental backups (having a backup from Monday, a backup from Wednesday...) then consider using rsnapshot. rsnapshot uses rsync and hardlinks to make minimal incremental backups. ...this isn't necessary, but you may be interested if it's something you'd never heard of before.

is there a way to verify that the backup performed by rsync is correct and usable?

rsync will tell you if there are any errors at the time the command finishes. If this is being run in something like a cronjob, you can tell it to notify you in some way (email...) if there are any errors. rsync does checksum checking, so I trust it.

Other than that, the only way to verify that the backup is usable is to try using it!

For fun, here is the rsync command I'm using now, in a file called "create-backup".

(WARNING: Do NOT run this script without understanding EVERYTHING in it. This script is tailored for MY specific computer and may delete files on another person's computer.)

#!/usr/bin/env bash

# The first command line argument will be used as the backup path destination
BACKUP_PATH="${1}"

# Confirm the destination directory exists
if [ -d "${BACKUP_PATH}" ]; then

    # Create a backup of my mariadb database, for my blog
    /storage/backups/archsystem/mariadb/create-backup

    # Create a list of the Arch Linux packages installed - there's no need
    # to save the entire operating system, just what packages I need!
    /storage/backups/archsystem/packages/create-backup

    # /etc
    # Backup the operating system configuration
    rsync -Cav --delete \
        /etc "${BACKUP_PATH}"

    # /home
    # The "excluded" directories are duplicated and are already backed
    # up via another directory path
    rsync -Cav --delete \
        --exclude=/home/drcouzelis/Music \
        --exclude=/home/drcouzelis/Pictures \
        --exclude=/home/drcouzelis/Shows \
        --exclude=/home/drcouzelis/Videos \
        /home "${BACKUP_PATH}"

    # /storage
    # Don't backup "Shows", it's huge and I don't care that much about it
    rsync -Cav --delete \
        --exclude=/storage/drcouzelis/Shows \
        /storage "${BACKUP_PATH}"

fi

I don't see a need to backup my entire operating system. Re-installing Arch Linux isn't that hard, especially if I save my "/etc" directory and the list of packages I have installed.

...lastly, don't forget the old saying, if you don't have at least one local backup and at least one off-site backup, YOU DON'T HAVE A BACKUP!

Edit: The official unofficial "backups" discussion thread is here, for more fun stuff to read: https://bbs.archlinux.org/viewtopic.php?id=141781

Last edited by drcouzelis (2018-09-12 12:53:55)

skunktrader

Member

From: Brisbane, Australia

Registered: 2010-02-14

Posts: 1,681

Re: Looking for the best way to make the system backup by means of rsync

If you haven't already evaluated it, have a look at rdiff-backup also.

micman

Member

Registered: 2016-08-10

Posts: 16

Re: Looking for the best way to make the system backup by means of rsync

So, it looks like the only difference between your "first time" and "incremental backups" command is the "--delete" argument? If that's the case, there's no need to ever use the "first time" command. Both commands will behave the same if the "/path/to/backup/folder" is initially empty.

Yes, I know, it’s ridiculous... but I wanted to show the steps that I had in mind to make a full system backup. Why full, or rather "full"?
I am in agreement with you that re-installing Arch Linux isn't hard. I didn't think it before to try. But also if I backup my "/etc" and list all the packages I have installed, I have to reinstall and, in particular, configure these last. This takes me much time. For example, after this installation of Arch Linux I have installed about sixty packages and all my personal documents are stored apart from the OS partition. Linux or Windows, I make it since long ago. So, also if I have a big trouble with the OS, I have more possibilities to have my data saved than they would be in the /home directory, for example. But this fact implies that all my mails, for example,  are stored in an other partition. So, when I reinstall the mail client--thunderbird in my case-- I have to reconfigure the local directories to store the mails and other parameters. The same is true for other programs: I use Firefox with relatively many add-ons; I use four or five Windows softwares--two essentially--by using Wine, and therefore I must reconfigure this last and them, and so on.

...lastly, don't forget the old saying, if you don't have at least one local backup and at least one off-site backup, YOU DON'T HAVE A BACKUP!

Exactly! For this reason I have a partition on an external HD where I backup all my documents, mails, etc. which are stored in the separated partition of my first HD nearby the Arch Linux partition. To backup them periodically, I use a software like FreeFileSync, relatively reliable and fast for this kind of backup, and a part of them is backuped a second time on a cloud (I use pCloud, that I have found that it runs well on Arch Linux). To backup the OS I use rsync (I'll take a look at rsnapshot in the future and with care to your script. Also ccrypt looks interesting for security reasons).

Other than that, the only way to verify that the backup is usable is to try using it!

Of course, but how to do it? If the installed system run well, you don't take the risk to try the backup superseding it and in case of a malfunction to find yourself without operating system. The other possibility is that you try the copy of the system on the device/partition where it's located. But, to do this, it's necessary to change the startup configuration, i.e. grub or an other boot loader configuration. Or I am wrong?
To finish, thank you for the necrobump reference... It isn't very orthodox, but sometimes it can be useful 

have a look at rdiff-backup also.

I'll do it.

Last edited by micman (2018-09-13 11:23:07)

drcouzelis

Member

From: Connecticut, USA

Registered: 2009-11-09

Posts: 4,092

Website

Re: Looking for the best way to make the system backup by means of rsync

Yes, I know, it’s ridiculous... but I wanted to show the steps that I had in mind to make a full system backup. Why full, or rather "full"?
I am in agreement with you that re-installing Arch Linux isn't hard. I didn't think it before to try. But also if I backup my "/etc" and list all the packages I have installed, I have to reinstall and, in particular, configure these last. This takes me much time. For example, after this installation of Arch Linux I have installed about sixty packages and all my personal documents are stored apart from the OS partition. Linux or Windows, I make it since long ago. So, also if I have a big trouble with the OS, I have more possibilities to have my data saved than they would be in the /home directory, for example. But this fact implies that all my mails, for example,  are stored in an other partition. So, when I reinstall the mail client--thunderbird in my case-- I have to reconfigure the local directories to store the mails and other parameters. The same is true for other programs: I use Firefox with relatively many add-ons; I use four or five Windows softwares--two essentially--by using Wine, and therefore I must reconfigure this last and them, and so on.

That's fine. If you have the time and the hard drive space and it's easy, then there's no problem backing up the entire operating system installation.

Also ccrypt looks interesting for security reasons

Haha I use "ccrypt" too! It's super easy and I think it is very secure. I use it to encrypt my files before uploading them to my school SSH server.

Other than that, the only way to verify that the backup is usable is to try using it!

Of course, but how to do it? If the installed system run well, you don't take the risk to try the backup superseding it and in case of a malfunction to find yourself without operating system. The other possibility is that you try the copy of the system on the device/partition where it's located. But, to do this, it's necessary to change the startup configuration, i.e. grub or an other boot loader configuration. Or I am wrong?

I guess I forgot to say the OTHER old saying, "If you can't restore your backup then you DON'T have a backup".

Yes, it's a problem. You can buy another hard drive and try to do a restore. Or if your hard drive is big enough, try to restore inside a virtual machine.

If you encrypt your backups, at least make sure you test DEcrypting your backups.

I haven't tested my current backup and restore procedure, because I think my backup plan is very simple. I trust that rsync copied the files correctly to the backup hard drive, and I can read the files on the backup hard drive at any time. I know I can re-install Arch Linux by following the same instructions I followed the first time.

The most important files on a personal computer (not a business server) are your personal files. Operating systems can always be re-installed, even if it is difficult. But you can't "re-create" wedding photos, or baby first-step videos, or entire software projects... So if I can still read my backup files, then I'm all set.

...actually, maybe I'll try creating a backup image of my operating system and saving it to my backup hard drive... Since I now have the space for it.

Last edited by drcouzelis (2018-09-13 13:38:31)

micman

Member

Registered: 2016-08-10

Posts: 16

Re: Looking for the best way to make the system backup by means of rsync

You can buy another hard drive and try to do a restore. Or if your hard drive is big enough, try to restore inside a virtual machine.

It isn’t necessary if you have installed Grub. In this case, with Super Grub Disk you can detect, test and, if it's good, start the Arch copy. It detects all kernels and operating systems on your connected devices. Choosing the kernel of your backup it gives you an error message, but after it starts directly the "new" system. This night I have made the backup and now I'm using this latter.
In any case, even if you install the system backup on a new device and you don't use Super Grub, if you want to test it you need to change the Grub configuration, at least temporarily, i.e. the device where the system backup is installed (for example from 'hd0, msdos1' to 'hd1, msdos1') and its UUID. You can find the informations you need with GParted and verify by means of it and/or dmesg what copy of Arch (now you have not one, but two copies ) is running. Of course, to verify the backups of your photos, emails and other personal documents, theses instructions aren't necessary.