You are not logged in.
- Topics: Active | Unanswered
#1 2024-01-29 21:38:51
- bepaald
- Member
- Registered: 2016-10-08
- Posts: 23
[SOLVED] Rsync not copying files: 'uptodate' though they don't exist
I have a RPI4 which I use as a media player. A portable hard drive is connected to that RPI4 which I usually copy media files to using `rsync` from my main desktop.
Today I noticed not all files get transferred. To troubleshoot, I've connected the drive directly to my PC (so no networking involved). The drive is ntfs formatted, so it does not support unix file permissions.
[~] $ # the source directory:
[~] $ ll /mnt/Media/Series/Bobs\ Burgers/
total 1091216
-rw-r--r-- 1 bepaald bepaald 218666559 Apr 17 2023 Bobs.Burgers.S13E18.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 bepaald bepaald 221363380 May 1 2023 Bobs.Burgers.S13E19.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 bepaald bepaald 227777867 May 8 2023 Bobs.Burgers.S13E20.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 bepaald bepaald 227720818 May 26 2023 Bobs.Burgers.S13E21.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 bepaald bepaald 221865639 May 26 2023 Bobs.Burgers.S13E22.WEBRip.x264-ION10.mp4
[~] $ # the target directory (local):
[~] $ ll /var/run/media/bepaald/MediaBox/Series/Bobs\ Burgers/
total 652160
-rw-r--r-- 1 root root 218666559 Apr 17 2023 Bobs.Burgers.S13E18.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 root root 221363380 May 1 2023 Bobs.Burgers.S13E19.WEBRip.x264-ION10.mp4
-rw-r--r-- 1 root root 227777867 May 8 2023 Bobs.Burgers.S13E20.WEBRip.x264-ION10.mp4
[~] $ # just a dry-run:
[~] $ rsync -n -t -r --size-only -vv -P /mnt/Media/Series/Bobs\ Burgers /var/run/media/bepaald/MediaBox/Series/
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
Bobs Burgers/Bobs.Burgers.S13E18.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E19.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E20.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E21.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E22.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 305 bytes received 457 bytes 1.524,00 bytes/sec
total size is 1.117.394.263 speedup is 1.466.396,67 (DRY RUN)
[~] $ Notice it says "is uptodate" for each file, even though 2 of the files do not exist on the target. There are some options to rsync which cause the files to be copied (like `-g`, because the file ownership and permissions differ), but those cause all files to be copied instead of just the new ones. If I just touch some other new file, it does seem to work for that:
[~] $ touch /mnt/Media/Series/Bobs\ Burgers/A_NEW_FILE
[~] $ rsync -n -t -r --size-only -vv -P /mnt/Media/Series/Bobs\ Burgers /var/run/media/bepaald/MediaBox/Series/
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
Bobs Burgers/Bobs.Burgers.S13E18.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E19.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E20.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E21.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/Bobs.Burgers.S13E22.WEBRip.x264-ION10.mp4 is uptodate
Bobs Burgers/
Bobs Burgers/A_NEW_FILE
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 357 bytes received 460 bytes 1.634,00 bytes/sec
total size is 1.117.394.263 speedup is 1.367.679,64 (DRY RUN)Why would rsync claim files that do not exist are 'uptodate' (regardless of size,hash,mtime or permissions)? Does anyone know how to fix this?
Thanks!
Last edited by bepaald (2024-01-30 13:53:12)
Offline
#2 2024-01-30 07:38:09
- d.ALT
- Member
- Registered: 2019-05-10
- Posts: 959
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
--ignore-existing skip updating files that exist on receiver
(get rid of -t and --size-only)
<49,17,III,I> Fama di loro il mondo esser non lassa;
<50,17,III,I> misericordia e giustizia li sdegna:
<51,17,III,I> non ragioniam di lor, ma guarda e passa.
Offline
#3 2024-01-30 07:54:19
- bepaald
- Member
- Registered: 2016-10-08
- Posts: 23
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
Thanks for your suggestion.
--ignore-existing skip updating files that exist on receiver
I don't really get it, the files don't exist on receiver, and the problem is that they are skipped already without this option.... Anyway:
[~] $ rsync -n -r --ignore-existing -vv -P /mnt/Media/Series/Bobs\ Burgers /var/run/media/bepaald/MediaBox/Series/
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
Bobs Burgers/Bobs.Burgers.S13E18.WEBRip.x264-ION10.mp4 exists
Bobs Burgers/Bobs.Burgers.S13E19.WEBRip.x264-ION10.mp4 exists
Bobs Burgers/Bobs.Burgers.S13E20.WEBRip.x264-ION10.mp4 exists
Bobs Burgers/Bobs.Burgers.S13E21.WEBRip.x264-ION10.mp4 exists
Bobs Burgers/Bobs.Burgers.S13E22.WEBRip.x264-ION10.mp4 exists
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 291 bytes received 417 bytes 1.416,00 bytes/sec
total size is 1.117.394.263 speedup is 1.578.240,48 (DRY RUN)So, now it says explicitly the files exist, even though they don't (which was already implied by "is uptodate" I suppose).
Last edited by bepaald (2024-01-30 07:57:23)
Offline
#4 2024-01-30 08:02:43
- d.ALT
- Member
- Registered: 2019-05-10
- Posts: 959
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
Thanks for your suggestion.
--ignore-existing skip updating files that exist on receiver
I don't really get it, the files don't exist on receiver, and the problem is that they are skipped already without this option
There are some options to rsync which cause the files to be copied (like `-g`, because the file ownership and permissions differ), but those cause all files to be copied instead of just the new ones
Anyway, what if:
rsync -vP /mnt/Media/Series/Bobs\ Burgers/* /var/run/media/bepaald/MediaBox/Series/Bobs\ Burgers/Can you please post here output of:
$ mount
# lsblk -f<49,17,III,I> Fama di loro il mondo esser non lassa;
<50,17,III,I> misericordia e giustizia li sdegna:
<51,17,III,I> non ragioniam di lor, ma guarda e passa.
Offline
#5 2024-01-30 08:09:14
- bepaald
- Member
- Registered: 2016-10-08
- Posts: 23
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
rsync -vP /mnt/Media/Series/Bobs\ Burgers/* /var/run/media/bepaald/MediaBox/Series/Bobs\ Burgers/
I had actually tried that, but the results are the same.
[~] $ rsync -nvP /mnt/Media/Series/Bobs\ Burgers/* /var/run/media/bepaald//MediaBox/Series/Bobs\ Burgers/
sent 237 bytes received 12 bytes 498,00 bytes/sec
total size is 1.117.394.263 speedup is 4.487.527,16 (DRY RUN)Can you please post here output of:[...]
[~] $ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=16318572k,nr_inodes=4079643,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p2 on / type ext4 (rw,relatime,lazytime,discard)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9049)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16333144k,nr_inodes=1048576,inode64)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /mnt/6TDATA type autofs (rw,relatime,fd=48,pgrp=1,timeout=240,minproto=5,maxproto=5,direct,pipe_ino=4433)
systemd-1 on /mnt/Media type autofs (rw,relatime,fd=49,pgrp=1,timeout=240,minproto=5,maxproto=5,direct,pipe_ino=4439)
/dev/sda2 on /mnt/1TDATA type ext4 (rw,noatime,lazytime,discard)
/dev/sdb1 on /mnt/VMS type ext4 (rw,noatime,lazytime,discard)
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,lazytime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro,discard,x-gvfs-hide)
/dev/sda1 on /home type ext4 (rw,relatime,lazytime,discard,x-gvfs-hide)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266628k,nr_inodes=816657,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/mapper/luks-e48769e0-9bea-41ba-992b-0eb2c3aadf0d on /mnt/Traffic type ext4 (rw,nosuid,nodev,noexec,noatime,lazytime,user=bepaald,x-gvfs-hide)
/dev/sdg1 on /run/media/bepaald/MediaBox type ntfs3 (rw,nosuid,nodev,relatime,uid=1000,gid=1000,iocharset=utf8,uhelper=udisks2)
/dev/sdc1 on /mnt/Media type ext4 (rw,nosuid,nodev,noexec,noatime,lazytime,x-systemd.automount,x-systemd.idle-timeout=4min)[~] $ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ext4 1.0 HOME 3d129fa0-47dd-4bfb-be62-699f51698ce8 685,8G 20% /home
└─sda2 ext4 1.0 1TDATA 8b6dcb38-7bb6-4f91-8d8a-061c13c7f663 647,8G 29% /mnt/1TDATA
sdb
└─sdb1 ext4 1.0 VMS cb3b382d-ac31-494b-b713-575f03167da2 95,1G 58% /mnt/VMS
sdc
└─sdc1 ext4 1.0 Media c848125a-28f0-47ec-a6f3-917b90f50507 5,7T 47% /mnt/Media
sdd
└─sdd1 ext4 1.0 6TDATA 88e6e281-96b4-44cf-96cf-106de11ebafa
sde
└─sde1 crypto_LUKS 2 e48769e0-9bea-41ba-992b-0eb2c3aadf0d
└─luks-e48769e0-9bea-41ba-992b-0eb2c3aadf0d ext4 1.0 Traffic 8217ec82-cc7d-4a99-8b7c-ff1e7a8be7b6 3,5T 68% /mnt/Traffic
sdf
└─sdf1 crypto_LUKS 2 6c7ecc77-0e7d-4119-bb1c-77abd0f3974a
sdg
└─sdg1 ntfs MediaBox 7E462B2F28B7AAA2 2,8T 38% /run/media/bepaald/MediaBox
nvme0n1
├─nvme0n1p1 vfat FAT32 EFI F1DB-867A 206,9M 31% /boot
└─nvme0n1p2 ext4 1.0 ROOT f717be4f-bbbf-4119-a602-c1b53ee7f14f 843,9G 3% /(edit: forgot to run that last one as root, but the output appears identical?)
Thanks again!
Last edited by bepaald (2024-01-30 08:11:10)
Offline
#6 2024-01-30 08:40:36
- V1del
- Forum Moderator
- Registered: 2012-10-16
- Posts: 25,231
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
ntfs3 is prone to slight file corruption on ntfs drives, chances are some file name index with that name exists and rsync getting that as an answer from the FS. You should try running a chkdsk from Windows and then check whether you can reproduce/test whether you have the same behaviour if you opt for ntfs-3g instead of ntfs3
Offline
#7 2024-01-30 13:52:38
- bepaald
- Member
- Registered: 2016-10-08
- Posts: 23
Re: [SOLVED] Rsync not copying files: 'uptodate' though they don't exist
Thank you, that was it. I ran chkdsk and it fixed errors, specifically mentioning these files. So, I'd say this is solved and fixed. But I'm going to bite the bullet anyway and reformat to a proper filesystem and recopy everything...
Thanks!
Offline