[SOLVED] Rsync as cp alternative, Btrfs CoW

Cvlc · 2021-12-07 17:29:42

Hi

is the following tip from Rsync#As cp/mv alternative valid for a btrfs system with CoW ?

To use sane defaults quickly, you could use some aliases:

function cpr() {
  rsync --archive -hh --partial --info=stats1,progress2 --modify-window=1 "$@"
} 
function mvr() {
  rsync --archive -hh --partial --info=stats1,progress2 --modify-window=1 --remove-source-files "$@"
}

i just tried it to move a video file and I was surprised that it took 3s, when it's instantaneous with cp -a

[edit] Well actually I tried the mvr function, so not sure this has anything to do with CoW.... but still surprised that it took 3s, doesn't feel like it's doing it right.

Last edited by Cvlc (2021-12-17 03:30:45)

Trilby · 2021-12-07 17:40:31

I have no btrfs experience, but if this move was within the same filesystem, then cp should be notably faster than rsync for the reasons clearly outlined in the section of the wiki you link to. Why would it surprise you that rsync took longer?

progandy · 2021-12-07 20:10:00

As of coreutils 9.0, cp does try to automatically use reflinks if the filesystem supports that. rsync does not support it: https://github.com/WayneD/rsync/issues/153
For mv it is similar, on the same filesystem it is possible to just move the inode reference into the new directory without having to move the file contents. It may be possible to use --link-dest with rsync to use hardlinks to simulate that behaviour, but I am not sure whether the combination --link-dest with --remove-source-files is supported.

Last edited by progandy (2021-12-07 20:11:10)

Cvlc · 2021-12-09 16:44:41

OK thanks. I'll look deeper into it but it does confirm that rsync is not a viable alternative to cp / mv on a btrfs system

Trilby · 2021-12-09 18:19:49

This should have nothing to do with btrfs. Using rsync in place of cp or mv to copy/move content within the same filesystem would be horrible on *any* filesystem. That's not what rsync is for - it's the wrong tool for the job.

Cvlc · 2021-12-14 16:47:57

Should the wiki be corrected then ?

Rsync#As cp/mv alternative

Trilby · 2021-12-14 16:54:35

I don't see why. That's a page all about rsync which is introduced from the start as a tool primarily used for transferring files between different machines (so different filesystems is implied) not advice for btrfs systems. While rsync can be used within a filesystem, and there may even be very good use cases for doing so, that's not the primary role of the tool. This is spelled out explicitly in that very section:

wiki wrote:

Files can be copied locally as with cp, but the motivating purpose of rsync is to copy files remotely, i.e. between two different hosts.

The 'r' in rysnc as well as the r in the cpr and mvr stand for remote. Taking tools intended for use between remote systems but using them within a filesystem and noting poor performance is a bit silly, and does not mean those tools don't have value. Just because hammers suck at driving screws doesn't mean the hardware store should stop selling hammers.

Last edited by Trilby (2021-12-14 16:57:52)

Cvlc · 2021-12-16 02:51:20

Well, I respectfully disagree :

The Core Utilities article has a cp_alternatives section which only says :

Using rsync#As cp/mv alternative allows you to resume a failed transfer, to show the transfer status, to skip already existing files and to make sure of the destination files integrity using checksums.

The link brings you to the rsync section which starts with

rsync can be used as an advanced alternative for the cp or mv command, especially for copying larger files:
$ rsync -P source destination

The part that you quoted says that it can be used locally, and does not mention at all that it's a bad idea to do so...

Like you said this has nothing to do with Btrfs, I use Rsync for backups and I know that it is essentially for remote systems, but when I read this I thought "oh yeah great idea".

It turns out not to be one which is why I think it's misleading. This thread wouldn't exist otherwise, and if I misunderstood others will.

The hardware store shouldn't stop selling hammers, but it shouldn't store the screwdrivers next to the nails and the hammers next to the screws....

thanks for your help

Last edited by Cvlc (2021-12-16 03:08:12)

Trilby · 2021-12-16 04:12:41

Ok, so you see a different page as misleading. In the case of the prior-existing content on the core utilities page, I'd completely agree. That was bad advice there and I see you've flagged it as disputed - though it's not just useful for the reasons stated for remote media, but anytime something is being moved between different filesystems. Resuming a within-filesystem move is a non-issue as such a move should be effectively instantaneous regardless of the size of the file - for a copy, on a non-COW/ROW filesystem there is at least some theoretical possibility of getting the benefits mentioned in that section of the core utilities wiki, but it still seems trivial and silly to me. But for a move it's completely nonsensical which is likely why a wiki admin moved it from a cp/mv alternative to just a cp alternative. That said, there is a bunch of nonsense on that page, which is kind of sad for a page for such an important package.

Last edited by Trilby (2021-12-16 04:31:13)

Cvlc · 2021-12-17 03:29:47

Thanks for the very interesting answers, I've updated the flag.

Have a good day !

Last edited by Cvlc (2021-12-17 03:30:24)

Arch Linux

#1 2021-12-07 17:29:42

[SOLVED] Rsync as cp alternative, Btrfs CoW

#2 2021-12-07 17:40:31

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#3 2021-12-07 20:10:00

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#4 2021-12-09 16:44:41

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#5 2021-12-09 18:19:49

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#6 2021-12-14 16:47:57

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#7 2021-12-14 16:54:35

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#8 2021-12-16 02:51:20

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#9 2021-12-16 04:12:41

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

#10 2021-12-17 03:29:47

Re: [SOLVED] Rsync as cp alternative, Btrfs CoW

Board footer