You are not logged in.

#1 2021-11-24 13:13:11

Cvlc
Member
Registered: 2020-03-26
Posts: 273

[SOLVED] How to corrrect timestamps modified by Rsync

Hi,

I have an rsync script to backup my data to my external drive when I plug it in. It 's a simple :

rsync -a --delete --backup --backup-dir="$ARCHIVE/$DATE" -f "merge $FILTERS" $SOURCE $DESTINATION

I recently installed Syncthing to sync files between my computer and my phone, and apparently in the process Syncthing updated the modification times of 20 Gb worth of pictures.
Now Rsync wants to copy all those pictures again to the backup HDD, even though they are the same, because the modification times differ.

Number of files: 30,811 (reg: 29,539, dir: 1,272)
Number of created files: 2,802 (reg: 2,738, dir: 64)
Number of deleted files: 114 (reg: 107, dir: 7)
Number of regular files transferred: 5,084
Total file size: 185,523,093,911 bytes
Total transferred file size: 23,957,034,587 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 65,532
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 709,625
Total bytes received: 23,064

sent 709,625 bytes  received 23,064 bytes  488,459.33 bytes/sec
total size is 185,523,093,911  speedup is 253,208.52 (DRY RUN)

How can I proceed with my backup without copying all those identical files, but without missing files which really have been modified ?

Thanks !

Last edited by Cvlc (2021-11-27 12:01:29)

Offline

#2 2021-11-24 13:28:57

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] How to corrrect timestamps modified by Rsync

You could possibly try passing the

--size-only

Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#3 2021-11-24 13:41:41

Cvlc
Member
Registered: 2020-03-26
Posts: 273

Re: [SOLVED] How to corrrect timestamps modified by Rsync

would that update files which have been modified but are the same size though ?

Offline

#4 2021-11-24 14:01:29

jonno2002
Member
Registered: 2016-11-21
Posts: 684

Re: [SOLVED] How to corrrect timestamps modified by Rsync

i highly doubt that files that have been modified would be the EXACT same size, i ran into this once and "--size-only" did the trick.

Offline

#5 2021-11-24 19:16:37

Cvlc
Member
Registered: 2020-03-26
Posts: 273

Re: [SOLVED] How to corrrect timestamps modified by Rsync

jonno2002 wrote:

i highly doubt that files that have been modified would be the EXACT same size, i ran into this once and "--size-only" did the trick.

What about a plain text file or a script ? If I ever modify a value to another of the same length, wouldn't that be the exact same size ? and then the file wouldn't copied over. Seems very risky to me.

Offline

#6 2021-11-24 20:36:33

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] How to corrrect timestamps modified by Rsync

Cvlc wrote:

What about a plain text file or a script ? If I ever modify a value to another of the same length, wouldn't that be the exact same size ? and then the file wouldn't copied over. Seems very risky to me.

You're moving away from what you were asking about in your OP.

If text files/scripts are that big of a risk to you, then just bite the bullet, and do the complete rsync.  Problem solved.


Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#7 2021-11-24 20:39:34

Cvlc
Member
Registered: 2020-03-26
Posts: 273

Re: [SOLVED] How to corrrect timestamps modified by Rsync

Thanks for your answer

Well my computer is a normal one and has both text files / scripts and media files.

To rephrase the question then, is there anyway to update modification times without doing the complete rsync ?

Offline

#8 2021-11-25 02:25:36

jonno2002
Member
Registered: 2016-11-21
Posts: 684

Re: [SOLVED] How to corrrect timestamps modified by Rsync

this seems to work:

#!/bin/bash

find $1 > /tmp/list_of_files

while IFS= read -r line 
do 
touch -r "$line" "${line/$1/$2}"
done < /tmp/list_of_files

rm /tmp/list_of_files

exit 0

so if the script is called "timefix" you would run

timefix /path/to/files_with_correct_times /path/to/files_to_correct_times

or in your case use the source and destination paths:

timefix $source $destination

hope that makes sense

[EDIT] updated script using bash variable substitution instead of sed !
[EDIT2] updated script to actually work when path/filenames have spaces in them..... SORRY my bad

Last edited by jonno2002 (2021-11-25 13:40:36)

Offline

#9 2021-11-25 13:50:01

Cvlc
Member
Registered: 2020-03-26
Posts: 273

Re: [SOLVED] How to corrrect timestamps modified by Rsync

Thanks very much, exactly what I was looking for !

I thought "touch" was only for creating empty files....

Will try it out and report back

[edit]
there's only a slight problem, it creates empty files for files that are not on the laptop anymore. But that's pretty easy to fix

Other than that it worked, thanks !

Last edited by Cvlc (2021-11-25 16:16:02)

Offline

#10 2021-11-26 01:38:24

jonno2002
Member
Registered: 2016-11-21
Posts: 684

Re: [SOLVED] How to corrrect timestamps modified by Rsync

o right well heres an update to the script which uses a list of files generated from the destination instead of the source and checks to make sure the same file exists in the source before updating the timestamp, and also lets you know if the file doesnt exist and cannot update the timestamp.

#!/bin/bash

find $2 > /tmp/list_of_files

while IFS= read -r line 
do
[ -f "${line/$2/$1}" ] || [ -d "${line/$2/$1}" ] && touch -r "${line/$2/$1}" "$line" || echo "${line/$2/$1}" "doesnt exist in the source path so cannot update timestamp"
done < /tmp/list_of_files

rm /tmp/list_of_files

exit 0

i have tested it and it seems to work fine but please let me know if i messed somthing up and ill fix it.

[EDIT] sorry i messed it up again, its fixed now, it was updating the timestamps the wrong way !

Last edited by jonno2002 (2021-11-26 01:46:58)

Offline

#11 2021-11-27 12:00:14

Cvlc
Member
Registered: 2020-03-26
Posts: 273

Re: [SOLVED] How to corrrect timestamps modified by Rsync

Great thank you very much !

have a great day

Offline

Board footer

Powered by FluxBB