Clarifications on full system backup using dd

maldus · 2017-12-02 12:39:18

Hello everyone,
I am trying to set up my laptop to have a (possibly bootable) full system backup using the dd tool, and I would like to confirm that I understand everything correctly before starting.

I currently have 3 drives: two 250G SSDs (for my OSs, arch linux being one of them) and one 1T HDD (for storage). I want to backup one of the SSDs into part of the HDD.

1. If I dd my home partition into the HDD will it be bootable by simply running grub-mkconfig? Here https://bbs.archlinux.org/viewtopic.php?id=219799 the first command suggests there might be problems with separate kernel versions (the HDD should be a backup so possibly with an older kernel), but I just don't understand how.
2. Kinda obvious, but just to be sure: it's not a good idea to dd from my system (while mounted), right?
3. How big should the backup partition be? Exactly the same size, or is it better a few GB larger just to be sure? Keep in mind that I want to use the rest of the HDD as storage.

Some output for reference:

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
dev              8086564        0   8086564   0% /dev
run              8092148     1200   8090948   1% /run
/dev/nvme0n1p2 244568380 64056060 168019264  28% /
tmpfs            8092148    16444   8075704   1% /dev/shm
tmpfs            8092148        0   8092148   0% /sys/fs/cgroup
tmpfs            8092148        8   8092140   1% /tmp
/dev/nvme0n1p1    523248    50360    472888  10% /boot
tmpfs            1618428       32   1618396   1% /run/user/1000

$ lsblk 
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 931.5G  0 disk 
sdb           8:16   0 232.9G  0 disk 
├─sdb1        8:17   0   450M  0 part 
├─sdb2        8:18   0    99M  0 part 
├─sdb3        8:19   0    16M  0 part 
└─sdb4        8:20   0 232.3G  0 part 
nvme0n1     259:0    0 238.5G  0 disk 
├─nvme0n1p1 259:1    0   512M  0 part /boot
└─nvme0n1p2 259:2    0   238G  0 part /

Where nvme0n is where Arch lives, the HDD is sda and sdb has Windows in it.

loqs · 2017-12-02 13:19:27

Do you intend to have the backup HDD installed at the same time as two NVME's?
As after cloning the NVME you will have created partitions and filesystems with duplicate UUID's which could confuse the boot-loader / kernel.
If may also create a duplicate ESP if the system is using EFI.

maldus · 2017-12-02 17:27:29

Do you intend to have the backup HDD installed at the same time as two NVME's?

Yes; probably (most likely) not a particularly useful setup, but I'd like to try it.

As after cloning the NVME you will have created partitions and filesystems with duplicate UUID's which could confuse the boot-loader / kernel.
If may also create a duplicate ESP if the system is using EFI.

So I should also care to change the destination partition's UUIDs?

Trilby · 2017-12-02 17:37:58

dd is the wrong tool for your goals. If you use dd the UUIDs will be duplicated, you can't change them.

Further, if you use dd and target the the 1TB HDD as the output, anything previously on the 1TB drive will be lost (in theory some may be recoverable but not easily: the partition table will be overwritten).

You could potentially dd partitions to partitions, but generally dd is intended to duplicate devices in which case anything previously on the target device is overwritten earning dd the nickname 'disk destroyer'. dd is very handy for it's intended purpose, but what makes you think it's the right tool for your job? It sounds like your goals would be much better acheived with rsync.

maldus · 2017-12-02 17:58:00

Further, if you use dd and target the the 1TB HDD as the output, anything previously on the 1TB drive will be lost (in theory some may be recoverable but not easily: the partition table will be overwritten).

Yes, I wanted to clone partitions.

what makes you think it's the right tool for your job? It sounds like your goals would be much better acheived with rsync.

It just seemed simpler than rsync, but of course I'm not sure about anything. To use rsync I should create a ext4 partition and copy there my file system as described in https://wiki.archlinux.org/index.php/Rs … up_utility, am I right? Would it then be recognized by grub as a different OS?

Last edited by maldus (2017-12-02 17:58:34)

loqs · 2017-12-02 19:52:41

The linux NVME has two parititons / filesystems on it. If the system is EFI based then / would be the ext4 and /boot would be fat32.
If you wanted to boot from the HDD without the NVME present you need /boot to again be fat32 assuming the system is EFI based.
/ being ext4 would be fine. rsyncing the contents of both / and /boot I do not see a problem with however you do not want the fat32 partition
on the HDD to be marked as an ESP while the NVME is present to avoid the firmware becoming confused.
When you wanted to switch the HDD to the become the boot device I expect you would need to boot from live media,
change the partition type of the fat32 partition on the HDD to mark it as the ESP, run grub-mkconfig to update grub's config.
Edit:
Another issue you may want to consider is you have a full backup but it is in the same location as the device it is backing up
so both could be destroyed by a common incident. A similar principal applies if you have rsync scheduled to automatically
perform backups, files are altered / removed on the master then the change is copied by rsync to the backup preventing recovery.

Last edited by loqs (2017-12-02 20:07:14)

ngoonee · 2017-12-02 20:03:44

I question whether a partition clone is ever the 'right' way of backing things up. Seems like it would be unacceptably slow (especially as this is a laptop and it won't be switched on all the time). What sort of laptop has 3 discs though?

dd is useful for duplicating partitions/disks. This means you are ALSO duplicating your empty space, your cache files, and everything else. The benefit is that, if done right, you can just restore using dd and have things back up and running.

rsync is more suitable for backing up data (you know, the only thing that really matters). This results in much more compact backups, and a smoother backup experience (only back up changed files, which is very helpful for continuous backup, even though the initial backup would be slower). Restoring is more complex, my restore plan is basically to reinstall Arch and then restore data using rsync again. I also back up my /etc which would help getting things set up. You could also backup your pacman package list for reinstallation (why backup the actual packages when they're so easy to reobtain).

Xyne · 2017-12-02 20:34:33

dd is not the right tool for this. Aside from wasting a lot of time and read-write cycles, you will overwrite your entire backup ever time you run dd. If something goes wrong, you may wipe everything in one go.

If you really just want single copies of all of your files on the backup, create a partition with the same size as your home partition (with whatever filesystem you want that supports the attributes that you want to preserve, such as timestamps, ownership, hardlinks, acls, etc.) and then use rsync.

If you want to keep deleted and modified versions of files then there are ways to do that with rsync too, but at that point you may as well just go for a proper and tested backup solution instead of rolling your own. Consider using something such as borg which is easy to set up and offers options such as compression, encryption, automatic pruning (e.g. keep old versions for a fixed period of time) and easy retrieval (e.g. you can mount any backup as a filesystem and use find/grep/cp/rsync/whatever to find and retrieve old versions of files).

maldus · 2017-12-03 08:20:11

Another issue you may want to consider is you have a full backup but it is in the same location as the device it is backing up
so both could be destroyed by a common incident. A similar principal applies if you have rsync scheduled to automatically
perform backups, files are altered / removed on the master then the change is copied by rsync to the backup preventing recovery.

I understand. In my mind I wanted to be able to boot arch no matter what if I screw something up (say, I back it up once a month and at some point I break my arch installation). However I now realize it just doesn't make sense, or rather it would make sense for me if it was as simple as dding my partition into the HDD, which I now see it isn't.

If you wanted to boot from the HDD without the NVME present you need /boot to again be fat32 assuming the system is EFI based.

Partially unrelated to my original question: what if I wanted to do it WITH the NVME present? To be more precise, at this point I'd like to have another linux installation into the HDD; something to use instead of a live usb, just for convenience. The GRUB wiki page (https://wiki.archlinux.org/index.php/GRUB) says every drive needs an EFI partition: do I need another /boot in the HDD? But isn't GRUB already in my NVME's /boot?

What sort of laptop has 3 discs though?

Mine I guess two OSes (completely separated, to make sure windows does not mess up anything) and some storage. I had one more slot and HDD are relatively cheap this day so I wanted to experiment a little as well.

rsync is more suitable for backing up data (you know, the only thing that really matters)

I see your point there.

but at that point you may as well just go for a proper and tested backup solution instead of rolling your own

I will probably end up doing that.

Thanks to everyone; I needed this clarification since what I was thinking about was stupid unconventional and I couldn't find my answers on Google/Wiki. Now I understand better.

Last edited by maldus (2017-12-03 08:48:00)

Trilby · 2017-12-03 11:50:35

maldus wrote:

In my mind I wanted to be able to boot arch no matter what if I screw something up (say, I back it up once a month and at some point I break my arch installation).

Then just keep a bootable iso on a flash drive.

This allows you to always have the tools to boot, chroot, and do any necessary repairs. Rolling back to a cloned system at the first sign of trouble is one sure way to ensure that you never learn how to avoid that trouble again in the future.

Don't seek do-overs. Seek knowledge.

Arch Linux

#1 2017-12-02 12:39:18

Clarifications on full system backup using dd

#2 2017-12-02 13:19:27

Re: Clarifications on full system backup using dd

#3 2017-12-02 17:27:29

Re: Clarifications on full system backup using dd

#4 2017-12-02 17:37:58

Re: Clarifications on full system backup using dd

#5 2017-12-02 17:58:00

Re: Clarifications on full system backup using dd

#6 2017-12-02 19:52:41

Re: Clarifications on full system backup using dd

#7 2017-12-02 20:03:44

Re: Clarifications on full system backup using dd

#8 2017-12-02 20:34:33

Re: Clarifications on full system backup using dd

#9 2017-12-03 08:20:11

Re: Clarifications on full system backup using dd

#10 2017-12-03 11:50:35

Re: Clarifications on full system backup using dd

Board footer