[Solved] Tools for cloning an arch installation

Jens Clasen · 2020-04-12 09:12:44

Good morning everyone!

I'm currently planning to move an arch installation from one laptop to another. While doing that, I plan to change the partitioning scheme and filesystem.

Traditionally, I would use tar and netcat for such an occasion or possibly even rsync. Since tar + nc worked pretty well, the last few times I did that, I initially thought, that's what I would do now, as well.

Thinking about that, that is a pretty old school approach, though, and I'm wondering, if there's something better available, nowadays.

I'm not asking you to take a lot of time with this question, but if you could drop some names, keywords or links for me to evaluate, that would be nice. Most of the googling I did, only found articles which cloned the whole partition and others were primarily talking about using rsync. Can you think of anything else to consider?

Thanks in advance for your efforts.

Regards, Jens

Last edited by Jens Clasen (2020-04-19 11:18:32)

graysky · 2020-04-12 12:41:27

You only need rsync for this task.

Trilby · 2020-04-12 12:55:37

Jens Clasen wrote:

... that is a pretty old school approach, though...

You mean it's stood the test of time?

There certainly are countless new, shiny, not-thoroughly tested methods that offer countless additional (useless) features (and openings for bugs) if you'd really like.

Is there a limitation to the tar/rsync approachs you are familiar with? Is there some featur you'd like that they lack? If so, describe what improvement you are seeking.

Or are you just looking for more complexity for complexity's sake and an overall buggier approach? If so, try any of these.

Personally I'd use rsync.

Jens Clasen · 2020-04-13 06:50:04

Good morning, once again!

And thank you for taking the time to comment on this thread.

I'm not really missing much. Beside - maybe - the fact that rsync or tar+nc are a bit slow.

Since rsync's primary purpose is syncing and dealing with increments, I'm not entirely convinced, that rsync is the optimal approach for the problem. At least speedwise, I'd expect tar + nc to be a bit faster, since the whole "is the file already available on the target"-question can be avoided for my multitude of files. I agree, though, that rsync's syntax is easier to work with, since there's only one tool involved. And: rsync is definitely more error tolerant, than the tar+nc approach. rsync does it's own checksums and it can resume a transfer after network loss.

In the end and probably, both approaches have there very own merits.

When I started this thread, I just wanted to see, if there's something new to learn here. The only known (at least to me) downside of the two discussed approaches is the overall downtime for both systems. While it's not strictly necessary to have both computers offline and do the transfer running a live cd on each end, that is the recommended way to go for such a transfer. (Yes, I know: a source system with most services disabled and a read-only mounted fs would work as well.) I just thought, there might be more current approaches, which might have a way to make a system snapshot and do the transfer from this snapshot, while the source system keeps running.

After reading a bit more about this, I'm pretty sure, I've started from a wrong expectation, though. Lvm would able to do the snapshots for me, therefor I should not need the cloning tool to do that for me. Unfortunately, the source system isn't setup with lvm and I'm not entirely sure, if it's really worth changing that, before doing the move. I will have to read a bit more about that, before making any decisions.

Regards, Jens

Trilby · 2020-04-13 12:35:03

I think your assessment of the two primary options is sound. But the important point to recognize when you say they may be slow, both rsync and tar+nc would be very efficient and snappy software. An exceedingly vast majority of the time it takes to move the data from one device to the other is due entirely to the limits of the hardware interface between the two devices (e.g., bandwidth). No software magic can change that limit.

The potential exception is compression. Rsync can compress on the fly which is why I may favor it over tar+nc (unless you were to add an additional compression tool, e.g., tar+lz+nc).

Your point on rsync checking the target device for the existence of the data is logically valid, but I think irrelevant. The time this takes is trivial compared to the time it takes to actually move the data. For big data transfers, this is also a nice insurance policy: I doubt tar+nc would be detectably faster (nor would tar+compression+nc be faster than rsync with compression), but if anything goes wrong at any point, having to restart from scratch would make the tar+nc approach massively slower.

All other tools that one might use for this task add "features" but don't change the underlying process. Software can't change the physical bandwidth available. In fact, quite a large number of the other options available just use rsync under the hood anyways. And while also an undetectable difference, I'm quite sure that rsync+flash-gui couldn't be faster than rsync.

NOTE: compression is not likely useful in your case. This would only be useful in cases where the bandwidth limiting step is substantially slower than the cpu-to-disk rate as compressed data would go through the narrow pipe, but all data would need to be uncompressed and written to disk over the cpu-to-disk rate anyways. If you do try to transfer between computers over a WAN, compression is almost certainly a good option, or over a WLAN it's most likely a good option. Over wired LAN I uncertain of the benefit. Over two disks that can just be plugged into the same machine, it's definitely not a good idea.

Last edited by Trilby (2020-04-13 12:39:24)

Jens Clasen · 2020-04-19 11:17:59

First of all: thank you for your comments tilby!

Given a 1Gbit cable connection, bandwidth wasn't the main concern for me, but yes, compression was something I used for such occasions in the past. Even tar can work with various compression levels out of the box.

These days a stable transfer was more my concern - maybe combined with more automatization. Excluding various caches, system directories and so on is something you have to remember. A snapshot at a given point in time and a workable pre-selection of directories would have been nice.

That said, I've finished the move yesterday and, following the recommendations from here and all over the net, I did it using rsync. The next time, I won't do that, though. rsync has some horribly indecent defaults regarding symbolic link handling. (And yes, I deactivated munging.) Unfortunately I've only learned that after making the first system update with the new system. Correcting that with rsync and new settings was therefor out of the question and I ended up manually repairing broken symbolic links with the help of some half decent scripting.

I admit, I certainly had to remember trilby saying:

There certainly are countless new, shiny, not-thoroughly tested methods that offer countless additional (useless) features (and openings for bugs) if you'd really like.

In the end, I wouldn't have needed the useless features of even rsync for my transfer, when I just wanted to copy a whole system, excluding the content (but not the directory) of directories named /proc, /sys, /dev, /tmp and everything with either a cache or a tmp in it's name.

I'm marking this thread as solved and I'm once again thanking you for your comments.

Best regards, Jens

Arch Linux

#1 2020-04-12 09:12:44

[Solved] Tools for cloning an arch installation

#2 2020-04-12 12:41:27

Re: [Solved] Tools for cloning an arch installation

#3 2020-04-12 12:55:37

Re: [Solved] Tools for cloning an arch installation

#4 2020-04-13 06:50:04

Re: [Solved] Tools for cloning an arch installation

#5 2020-04-13 12:35:03

Re: [Solved] Tools for cloning an arch installation

#6 2020-04-19 11:17:59

Re: [Solved] Tools for cloning an arch installation

Board footer