Arch Linux

nbd

Member

Registered: 2014-08-04

Posts: 389

[SOLVED] Question to rsync experts: -H option copies unmodified files.

Greetings,

when the daily system backup with rsync is performed, thousands files (e.g. from /usr directory) which are not modified are uselessly transferrred to the backup storage every day. The rsync command is the following:

rsync --update -DHAErlptgo --relative --atimes --delete-during ${SRCPATH} ${BACKUPDIR}

Quick investigation shown, that these files are hard links, and the command option related to this behaviour is -H: if to remove it, the unmodified files then are not copied. Seemingly this situation is described in the rsync manual:

If  incremental recursion is active (see --recursive), rsync may transfer a missing
hard-linked file before it finds that another link for that contents  exists  else‐
where  in  the  hierarchy.  This does not affect the accuracy of the transfer (i.e.
which files are hard-linked together), just its efficiency (i.e. copying  the  data
for a new, early copy of a hard-linked file that could have been found later in the
transfer in another member of the hard-linked set of files).  One way to avoid this
inefficiency  is  to  disable  incremental  recursion  using the --no-inc-recursive
option.

Adding --no-inc-recursive option to the aforementioned command line doesn't change anything (obviously when used with -r/--recursive option it's simply ignored). Using --dirs instead of --recursive doesn't sync subdirectories at all with that option set. --delete-after instead of --delete-during also doesn't help.

Does someone know how to prevent useless transfer of unmodified hard-linked files with rsync?

Thanks.

Last edited by nbd (2016-03-28 17:26:52)

---

bing different

x33a

Forum Fellow

Registered: 2009-08-15

Posts: 4,587

Re: [SOLVED] Question to rsync experts: -H option copies unmodified files.

Please show the output of your rsync command with the --itemize-changes option. Post it to some pastebin site.

nbd

Member

Registered: 2014-08-04

Posts: 389

Re: [SOLVED] Question to rsync experts: -H option copies unmodified files.

Below is a part of rsync output for /usr/bin. The command was:

rsync --itemize-changes --update -DHAErlptgo --relative --atimes --delete-during -n /usr ${BACKUPDIR}

Fragment of rsync output:

.L..t..g... /usr/bin/2to3 -> 2to3-3.5
.L..t...... /usr/bin/Thunar -> thunar
.L..t...... /usr/bin/X -> Xorg
hf......u.. /usr/bin/aclocal-1.15
.L..t...... /usr/bin/appletviewer -> /usr/lib/jvm/default/bin/appletviewer
.L..t...... /usr/bin/apropos -> whatis
.L..t...... /usr/bin/arecord -> aplay
.L..t..g... /usr/bin/assistant -> qtchooser
hf......u.. /usr/bin/automake-1.15
.L..t...... /usr/bin/avahi-browse-domains -> avahi-browse
.L..t...... /usr/bin/avahi-publish-address -> avahi-publish
.L..t...... /usr/bin/avahi-publish-service -> avahi-publish
.L..t...... /usr/bin/avahi-resolve-address -> avahi-resolve
.L..t...... /usr/bin/avahi-resolve-host-name -> avahi-resolve
.L..t...... /usr/bin/awk -> gawk
.L..t...... /usr/bin/bshell -> bssh
.L..t...... /usr/bin/btrfsck -> btrfs
.L..t...... /usr/bin/bunzip2 -> bzip2
.L..t...... /usr/bin/bvnc -> bssh
.L..t...... /usr/bin/bzcat -> bzip2
.L..t...... /usr/bin/cache_check -> pdata_tools
.L..t...... /usr/bin/cache_dump -> pdata_tools
.L..t...... /usr/bin/cache_metadata_size -> pdata_tools
.L..t...... /usr/bin/cache_repair -> pdata_tools
.L..t...... /usr/bin/cache_restore -> pdata_tools
.L..t...... /usr/bin/captoinfo -> tic
.L..t...... /usr/bin/cc -> gcc
.L..t..g... /usr/bin/clang -> clang-3.7
.L..t...... /usr/bin/clang++ -> clang
.L..t..g... /usr/bin/clang-cl -> clang
.L..t...... /usr/bin/ctstat -> lnstat
.f......u.. /usr/bin/dbus-launch
.L..t...... /usr/bin/debugfs.reiserfs -> debugreiserfs
.L..t...... /usr/bin/depmod -> kmod
.L..t..g... /usr/bin/designer -> qtchooser
.L..t...... /usr/bin/desktop-file-edit -> desktop-file-install
.L..t..g... /usr/bin/dmstats -> dmsetup
.L..t..g... /usr/bin/dosfsck -> fsck.fat
.L..t..g... /usr/bin/dosfslabel -> fatlabel
.L..t...... /usr/bin/edit -> ex
.L..t...... /usr/bin/enconv -> enca
.L..t...... /usr/bin/era_check -> pdata_tools
.L..t...... /usr/bin/era_dump -> pdata_tools
.L..t...... /usr/bin/era_invalidate -> pdata_tools
.L..t...... /usr/bin/era_restore -> pdata_tools
.L..t...... /usr/bin/eview -> vim
.L..t...... /usr/bin/evim -> vim
.L..t...... /usr/bin/extcheck -> /usr/lib/jvm/default/bin/extcheck
.L..t...... /usr/bin/firefox -> /usr/lib/firefox/firefox
.L..t...... /usr/bin/flex++ -> flex
.L..t...... /usr/bin/fr -> foxitreader
hf......u.. /usr/bin/fsck.ext2
hf......u.. /usr/bin/fsck.ext3
hf......u.. /usr/bin/fsck.ext4
hf......u.. /usr/bin/fsck.ext4dev
.L..t..g... /usr/bin/fsck.msdos -> fsck.fat
.L..t...... /usr/bin/fsck.reiserfs -> reiserfsck
.L..t..g... /usr/bin/fsck.vfat -> fsck.fat
hf......u.. /usr/bin/g++
hf......u.. /usr/bin/gawk-4.1.3
.f......u.. /usr/bin/gcc-ranlib
.L..t...... /usr/bin/geqn -> eqn
.L..t..g... /usr/bin/ghc -> ghc-7.10.3
.L..t..g... /usr/bin/ghc-pkg -> ghc-pkg-7.10.3
.L..t..g... /usr/bin/ghci -> ghci-7.10.3
.L..t...... /usr/bin/gimp -> gimp-2.8
.L..t...... /usr/bin/gimp-console -> gimp-console-2.8
.L..t...... /usr/bin/gimptool -> gimptool-2.0
hf......u.. /usr/bin/git-receive-pack
hf......u.. /usr/bin/git-upload-archive
.L..t...... /usr/bin/gksudo -> gksu
.L..t...... /usr/bin/gpg -> gpg2
.L..t...... /usr/bin/gpgv -> gpgv2
.L..t...... /usr/bin/gtbl -> tbl
.L..t...... /usr/bin/guile-tools -> guild
.L..t...... /usr/bin/gview -> vim
.L..t...... /usr/bin/gvim -> vim
.L..t...... /usr/bin/gvimdiff -> vim
.L..t..g... /usr/bin/haddock -> haddock-ghc-7.10.3
.L..t...... /usr/bin/halt -> systemctl
.L..t...... /usr/bin/hspell-i -> hspell
.L..t...... /usr/bin/i386 -> setarch
hf......u.. /usr/bin/i686-pc-linux-gnu-c++
hf......u.. /usr/bin/i686-pc-linux-gnu-g++
hf......u.. /usr/bin/i686-pc-linux-gnu-gcc
hf......u.. /usr/bin/i686-pc-linux-gnu-gcc-5.3.0
hf......u.. /usr/bin/i686-pc-linux-gnu-gcc-ar
hf......u.. /usr/bin/i686-pc-linux-gnu-gcc-nm
hf......u.. /usr/bin/i686-pc-linux-gnu-gcc-ranlib
.f......u.. /usr/bin/i686-pc-linux-gnu-pkg-config
.L..t...... /usr/bin/idle -> idle3
.L..t..g... /usr/bin/idle3 -> idle3.5
.L..t...... /usr/bin/idlj -> /usr/lib/jvm/default/bin/idlj
.L..t...... /usr/bin/infotocap -> tic
.L..t...... /usr/bin/init -> ../lib/systemd/systemd
.L..t...... /usr/bin/insmod -> kmod
.L..t...... /usr/bin/ip6tables -> xtables-multi
.L..t...... /usr/bin/ip6tables-restore -> xtables-multi
.L..t...... /usr/bin/ip6tables-save -> xtables-multi
.L..t...... /usr/bin/iptables -> xtables-multi
.L..t...... /usr/bin/iptables-restore -> xtables-multi
.L..t...... /usr/bin/iptables-save -> xtables-multi
.L..t...... /usr/bin/iptables-xml -> /usr/bin/xtables-multi
.L..t...... /usr/bin/jar -> /usr/lib/jvm/default/bin/jar
.L..t...... /usr/bin/jarsigner -> /usr/lib/jvm/default/bin/jarsigner
.L..t...... /usr/bin/java -> /usr/lib/jvm/default-runtime/bin/java
.L..t...... /usr/bin/javac -> /usr/lib/jvm/default/bin/javac
.L..t...... /usr/bin/javadoc -> /usr/lib/jvm/default/bin/javadoc
.L..t...... /usr/bin/javah -> /usr/lib/jvm/default/bin/javah
.L..t...... /usr/bin/javap -> /usr/lib/jvm/default/bin/javap
.L..t...... /usr/bin/jcmd -> /usr/lib/jvm/default/bin/jcmd
.L..t...... /usr/bin/jconsole -> /usr/lib/jvm/default/bin/jconsole
.L..t...... /usr/bin/jdb -> /usr/lib/jvm/default/bin/jdb
hf......u.. /usr/bin/jfs_fsck
.L..t...... /usr/bin/jhat -> /usr/lib/jvm/default/bin/jhat
.L..t...... /usr/bin/jinfo -> /usr/lib/jvm/default/bin/jinfo
.L..t...... /usr/bin/jmap -> /usr/lib/jvm/default/bin/jmap
.L..t...... /usr/bin/jps -> /usr/lib/jvm/default/bin/jps
.L..t...... /usr/bin/jrunscript -> /usr/lib/jvm/default/bin/jrunscript
.L..t...... /usr/bin/jsadebugd -> /usr/lib/jvm/default/bin/jsadebugd
.L..t...... /usr/bin/jstack -> /usr/lib/jvm/default/bin/jstack
.L..t...... /usr/bin/jstat -> /usr/lib/jvm/default/bin/jstat
.L..t...... /usr/bin/jstatd -> /usr/lib/jvm/default/bin/jstatd
.L..t...... /usr/bin/keytool -> /usr/lib/jvm/default-runtime/bin/keytool
.L..t...... /usr/bin/lastb -> last
.L..t..g... /usr/bin/lconvert -> qtchooser
hf......u.. /usr/bin/ld.bfd
/* the rest output is skipped */

One can note, that not all /usr/bin files are in this output. The pattern for missing files is that one member of hard linked set is missing, and other is marked with 'u' (for updating).  E.g. /usr/bin/ld.bfd is listed (and marked 'u'), but /usr/bin/ld - is not. Both have the same inode:

[root@kw usr]# find /usr/bin -inum 389670
/usr/bin/ld
/usr/bin/ld.bfd

The same is true for 'c++' set:

 [root@kw usr]# find /usr/bin -inum 403298
/usr/bin/g++
/usr/bin/i686-pc-linux-gnu-c++
/usr/bin/i686-pc-linux-gnu-g++
/usr/bin/c++

The first three files are listed and marked, the forth - is not.

---

bing different

frostschutz

Member

Registered: 2013-11-15

Posts: 1,658

Re: [SOLVED] Question to rsync experts: -H option copies unmodified files.

Maybe you should take this to the rsync mailing list.

Unnecessary transfers are understandable if some of the hard links are missing on the target and rsync simply tries to copy them first before discovering the file was already there under another name.

If it keeps transferring the same files every time you run the command even though they're already there - something else seems to be wrong, it might be a bug in rsync.

nbd

Member

Registered: 2014-08-04

Posts: 389

Re: [SOLVED] Question to rsync experts: -H option copies unmodified files.

Unnecessary transfers are understandable if some of the hard links are missing on the target and rsync simply tries to copy them first before discovering the file was already there under another name.

It seems something like this is just the case and even is described in rsync manual:

If  incremental recursion is active (see --recursive), rsync may transfer a missing
hard-linked file before it finds that another link for that contents  exists  else‐
where  in  the  hierarchy. 

The solution proposed in the manual is to use --no-inc-recursive, but I cannot see how it can be combined with --recursive option which I use for recursively backup/update all directory tree including subdirectories. (When both these options are used, the --no-inc-recursive seemingly doesn't have any effect.)

---

bing different

nbd

Member

Registered: 2014-08-04

Posts: 389

Re: [SOLVED] Question to rsync experts: -H option copies unmodified files.

It turned out that this is really a bug. Not in rsync itself though, but in one of the patches distributed from the official rsync download page.

---

bing different