Arch Linux

Salkay

Member

Registered: 2014-05-22

Posts: 663

rsync failing on the same file for different systems

I have three Arch systems, and all of them backup to my (Arch) server using backintime, which is rsync-based. All three backups are currently failing on the exact same file, /usr/share/licenses/lib32-libx11/COPYING (provided by lib32-libx11), with the following error

[I] Take snapshot (rsync: ERROR: usr/share/licenses/lib32-libx11/COPYING failed verification -- update discarded.)

I tried manually rsyncing the file, and it seems to copy fine. How could this file be causing a problem on all three systems? As far as I can see, the file is fine, but even if not I can't see how it could be consistently broken. Maybe a bug in rsync?

FWIW I tried to move the file away, and now the backups proceed past this point, but fail for /usr/share/licenses/lib32-libvorbis/COPYING (provided by lib32-libvorbis).

mountaintrek

Member

Registered: 2024-02-01

Posts: 48

Re: rsync failing on the same file for different systems

Maybe something like this: https://github.com/RsyncProject/rsync/issues/715

I would look at the attributes of the file and the options on the backintime rsync command.

Use journalctl -b to check the journal for backintime messages.

Check the last log in backintime gui (https://backintime.readthedocs.io/en/latest/log.html) for additional messages or return codes. rsync man page has some return codes listed under the EXIT VALUES section.

Also checkout backintime ISSUES: https://github.com/bit-team/backintime/issues

# The 32-bit directory is a symlink to the 64-bit
===> stat /usr/share/licenses/lib32-libx11
  File: /usr/share/licenses/lib32-libx11 -> libx11

Last edited by mountaintrek (2025-02-21 20:56:56)

Salkay

Member

Registered: 2014-05-22

Posts: 663

Re: rsync failing on the same file for different systems

Thanks @mountaintrek, much appreciated. I actually posted here because I thought it might have been an Arch packaging issue, although now that seems less likely.

I found the command used to backup in backintime, which is

[I] rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --acls --xattrs --perms --executability --group --owner
--info=progress2 --no-inc-recursive --numeric-ids --rsh=ssh -o
ServerAliveInterval=240 -o LogLevel=Error -o IdentityFile=/root/.ssh/root@pi_rsa
-p 706 -o Ciphers=aes128-ctr --delete --delete-excluded -v -i
--out-format=BACKINTIME: %i %n%L --link-dest=../../20250216-162501-629/backup
--chmod=Du+wx --exclude=/root/.local/share/backintime/mnt/1_291887
--exclude=/root/.local/share/backintime --exclude=.local/share/backintime/mnt
--exclude=/dev/* --exclude=/media/* --exclude=/mnt/* --exclude=/proc/*
--exclude=/sys/* --exclude=/tmp/* --exclude=/run/*
--exclude=/var/lib/pacman/sync/* --exclude=/home/*/mnt/* --exclude=/var/lock/*
--exclude=/home/*/.cache/* --exclude=/home/*/.local/share/Trash*/*
--exclude=/home/*/tmp/* --exclude=/var/cache/pacman/* --exclude=/backup/*
--exclude=/home/*/.decrypt --exclude=/home/*/.recoll/xapiandb/* --include=/
--include=/** --exclude=* /
root@192.168.1.91:/media/PiHDD2/work_laptop_backup/backintime/work-laptop/root/1/new_snapshot/backup

(I don't use the backintime GUI because it always seems to mess with the crontab. Instead I just read from ~/.local/share/backintime/takesnapshot_.log. I'll have a look at the other sources when I catch the error again, because it has since been removed.)

I can replicate the bug from the issue you linked, i.e. with rsync -KRlptzv. Oddly enough I can't replicate with the rsync options above (I remove non-relevant parts like --exclude and --link-dest.)

What did you mean with the symlink comment? What command created your code block output? On my system I don't think /usr/share/licenses/lib32-libx11/COPYING nor any parent directory is a symlink?

mountaintrek

Member

Registered: 2024-02-01

Posts: 48

Re: rsync failing on the same file for different systems

What did you mean with the symlink comment? What command created your code block output? On my system I don't think /usr/share/licenses/lib32-libx11/COPYING nor any parent directory is a symlink?

HHHmmm. I did an update recently and it appears the directory is no longer a symlink.

To confirm I did the following:

1. Searched for the package lib32-libx11 in archlinux online package database.
   https://archlinux.org/packages/multilib … 32-libx11/

2. On the right, clicked on Source Files, which goes to gitlab.

3. In the main body list, clicked on PKGBUILD. It's the last one in the list.

4. In the upper right, clicked on HISTORY.

5. Clicked on the "Feb 05, 2025 - upgpkg: 1.8.11-1: upstream update 1.8.11 " commit.

6. Scrolled down to the PKGBUILD changes to review the diff results.

45         - mkdir -p "$pkgdir/usr/share/licenses"
46         - ln -s $_pkgbasename "$pkgdir/usr/share/licenses/$pkgname"
     40    + install -d -m755 "${pkgdir}/usr/share/licenses/${pkgname}"
     41    + install -m644 COPYING "${pkgdir}/usr/share/licenses/${pkgname}/"
}

# Before update
===> ls -ld /usr/share/licenses/lib32-libx*
lrwxrwxrwx 1 root root    6 2024-08-06 17:23 /usr/share/licenses/lib32-libx11 -> libx11

# After update
===> ls -ld /usr/share/licenses/lib32-libx*
drwxr-xr-x 2 root root 4096 2025-02-22 19:48 /usr/share/licenses/lib32-libx11

backintime devs seem to be working on enhancing the reasons for a rsync error 23, with this issue
https://github.com/bit-team/backintime/issues/1587.  A combination of backintime issues and the backintime log, and journal should help you pinpoint the issue.

Last edited by mountaintrek (2025-02-24 07:28:56)

Salkay

Member

Registered: 2014-05-22

Posts: 663

Re: rsync failing on the same file for different systems

Thanks again @mountaintrek. I actually removed the files to get the backup working in the meantime, so I actually can't easily reproduce now. I'll post in this thread if a future update breaks it again.

backintime devs seem to be working on enhancing the reasons for a rsync error 23, with this issue

That's great to hear. I've been bitten before by broken updates persisting, and then subsequent backups taking up too much room (since they are no longer hard links). I'm aware that there's a backintime setting to not continue on errors, but this doesn't seem to work consistently. I'm glad they are looking to address this.

mountaintrek

Member

Registered: 2024-02-01

Posts: 48

Re: rsync failing on the same file for different systems

To help the archlinux moderators out...

If you're statisifed with the outcome, please prepend [Solved] to the thread title (edit first post)

Salkay

Member

Registered: 2014-05-22

Posts: 663

Re: rsync failing on the same file for different systems

To help the archlinux moderators out...

If you're statisifed with the outcome, please prepend [Solved] to the thread title (edit first post)

Thanks again. I don't think it's quite solved yet though.