Slow Sync Transfer Speeds [SOLVED]

dave1953 · 2023-09-03 18:18:53

I have, amongst my other drives a 3tb sata3 hdd and a 6tb sata3 hdd. These are used primarily to store media files, split by category across the two drives. These are duplicated by an exactly matching pair of external USB3 drives, same make, model and capacity in each case.

I use FreeFilesync to keep these four drives in sync as I add music/video files ripped from my own extensive cd and dvd collection to the source drives and back these up to the external drives. The aur version of FreeFilesync refused to compile correctly so i use the latest version of FreeFilesync downloaded from the FreeFilesync webpage.

These files are mainly .mkv files, flac files and .mp3 files. There are also some family videos in .avi format, photos in .jpeg and .odt files. I recently decided to use filebot and musicbrainz picard to ensure a strict name and numbering convention to all the media files. I was pleased at how quickly the renaming process was completed.

As usual I launched the FreeFilesync application to back my media files up to the backup drives and was shocked that 1) FreeFilesync completely rewrote the whole files as new ones rather than just changing the titling of them, 2) the transfer speed fell very quickly from the expected high speeds to a very poor 36mb/s, meaning that FreeFilesync is quoting 24 hours to transfer my media files, between 4 and 5tb.

Is this normal behaviour? Never known FreeFilesync take such a long time due to such a poor transfer speed. I am used to somewhere between 120mb/s to 300mb/s when I'm doing incremental syncs, so I'm concerned as to whether there is a configuration error or interface error as the transfer is so painfully slow. Or is it because FreeFilesync is insistent on completely rewriting the whole drives of .mkv, .avi, .flac, .mp3, .odt files rather than renaming the existing files. Thoughts and comments, please.

Last edited by dave1953 (2023-09-04 07:27:38)

seth · 2023-09-03 18:43:16

Thoughts and comments, please.

https://en.wikipedia.org/wiki/Paragraph

dave1953 · 2023-09-03 20:11:31

I am very familiar with paragraphs; however when I have created paragraphs in forums in the past the completed contribution has ignored my formatting and concenatated the text in a block format.

Apologies for not realising the Arch Linux Forum does in fact recognise paragraph formatting.

Perhaps I should have attempted formatting rather than assuming my efforts would not have been recognised.

jonno2002 · 2023-09-03 22:49:25

dave1953 wrote:

As usual I launched the FreeFilesync application to back my media files up to the backup drives and was shocked that 1) FreeFilesync completely rewrote the whole files as new ones rather than just changing the titling of them, 2) the transfer speed fell very quickly from the expected high speeds to a very poor 36mb/s, meaning that FreeFilesync is quoting 24 hours to transfer my media files, between 4 and 5tb.
Is this normal behaviour? Never known FreeFilesync take such a long time due to such a poor transfer speed. I am used to somewhere between 120mb/s to 300mb/s when I'm doing incremental syncs, so I'm concerned as to whether there is a configuration error or interface error as the transfer is so painfully slow. Or is it because FreeFilesync is insistent on completely rewriting the whole drives of .mkv, .avi, .flac, .mp3, .odt files rather than renaming the existing files. Thoughts and comments, please.

as far as freefilesync is concerned they are different files cause the names are different, its doing its job perfectly fine copying files over that dont yet exist. if you want to just change the file names you have to do that yourself on the destination files as well as the source using whatever program you used.

as for transfer speed, try copying or syncing a single large file and see what the speed is, copying lots of small files can slow transfer speeds considerably, or to rule out freefilesync try using cp or rsync instead just for testing purposes.

seth · 2023-09-04 04:45:47

I'm also not at all surprised that the sync copies the files instead of replicating your renaming, it cannot know the mapping and would have to unconditionally search the drives for matching checksums for deduplication, what's gonna be a rather involved process as well.

As for the speed: you'll get reported very high (reading) speeds as long as the file cache can be used and then the writing happens somewhat secretly (but unmounting the drive would take longer)
If you're copying so much that you're running out of cache and syncing dirty pages to disk, you'll drop down to the write speed of the destination.
=> What's the exact model of the destination drive?
You can also measure its raw performance using "dd if=/dev/zero oflag=direct of=/path/to/destination/file bs=8M count=128"

dave1953 · 2023-09-04 05:26:12

Thanks. The drive is reported as an ST6000NM0275-2BS110 (SF02)

dave1953 · 2023-09-04 07:09:46

[SOLVED] it's something to do with the motherboard usb3 socket. Got on my hands and knees, unplugged the connection and instead plugged it into the front usb3 connection. Going like a train now. Further investigation later.

Arch Linux

#1 2023-09-03 18:18:53

Slow Sync Transfer Speeds [SOLVED]

#2 2023-09-03 18:43:16

Re: Slow Sync Transfer Speeds [SOLVED]

#3 2023-09-03 20:11:31

Re: Slow Sync Transfer Speeds [SOLVED]

#4 2023-09-03 22:49:25

Re: Slow Sync Transfer Speeds [SOLVED]

#5 2023-09-04 04:45:47

Re: Slow Sync Transfer Speeds [SOLVED]

#6 2023-09-04 05:26:12

Re: Slow Sync Transfer Speeds [SOLVED]

#7 2023-09-04 07:09:46

Re: Slow Sync Transfer Speeds [SOLVED]

Board footer