Rsync, slow, failing to copy some files locally between external disks

jbodhorn · 2017-05-24 20:58:17

I'm noticing rsync failing to copy at least some files over and over, I'm copying locally between USB 3.0 hard drives hooked up to my laptop, a lenovo yoga 12 1st gen if that matters. I'm using a powered USB 3.0 hub for the non powered portable external drives that I'm copying from, they are all formatted NTFS and the hub is plugged into one of my two USB 3.0 ports. The new drive I'm copying to is a desktop external USB 3.0 drive that has it's own power source, it is formatted ext4 and plugged directly into the laptop's other USB 3.0 port.

Some of the files that fail are rather large, system images and such I've DD'd of laptops, desktops, RPi's and other SOC, but there also are a bunch of smaller files that I've seen fail also.

I just figured out there is an option to log rsync to a separate file and I'm trying it now to see if it is for sure the same files failing, I know at least one file, an image of a raspberry pi's sd card has failed to fully copy every time I've tried, I'm not yet sure if the other files that have failed have also been consistantly the same files.

I've been using the -P option and the files that fail always go down to transferring at 0MB/s then put out an I/O error, sometimes after that the rest of the files on the disk fail to copy, sometimes after that the rest try to copy and mostly succeed.

I looked in my rsync.conf for anything that could give me a clue why I keep getting fails but there really isn't much there, it's still the default 10 lines as it comes when you install it on Arch. I checked in the Archwiki but there isn't any sort of troubleshooting section for rsync, if anyone can point me in the right direction it'd be much appreciated.

Also, rsync has been kinda slow, the fastest I've seen it go is about 30MB/s on USB 3.0. Is this normal or should I be getting better transfer speeds? I was concluding this speed issue was because I'm going USB 3 external hdd to USB 3 external hdd, I wasn't sure if filesysytem plays a role in the speed in this case or if there could be another issue. I'm rsync'ing 3x 1tb + 1x 500gb NTFS formatted portable USB 3.0 hdd to a single 4tb ext4 formatted desktop hdd, any input would be great at the moment

jbodhorn · 2017-05-24 22:12:08

Here is the error I just got:

wd/RPiBplus-kodi-hotspot-vpn.img
 16,012,804,096 100%    2.29MB/s    1:51:02 (xfr#1, ir-chk=2113/462181)
rsync: read errors mapping "/media/wd/RPiBplus-kodi-hotspot-vpn.img": Input/output error (5)
WARNING: wd/RPiBplus-kodi-hotspot-vpn.img failed verification -- update discarded (will try again).

I think that's the first time it's copied all the way through though

jbodhorn · 2017-05-24 22:25:31

I should add that i have checked my disks for errors with windows check disk before attempting to use rsync and didn't get any errors. I scanned all the drives for bad sectors but that was a few months ago, maybe my drives are starting to fail and now have bad sectors. My external drives were getting old, that was the reason I bought the new 4tb drive, but last time i checked they were still in good health. I'm going to use my log to make an exclude list and copy what I can, then check once again for bad sectors

I tried to use plain old cp to copy that file I posted the error above for, failed 2x

sudo cp /media/wd/RPiBplus-kodi-hotspot-vpn.img /mnt/SEAGATE/wd/
cp: error reading '/media/wd/RPiBplus-kodi-hotspot-vpn.img': Input/output error

Slithery · 2017-05-24 22:38:19

Run a SMART test on the source drive and post the results. This sounds like a failing drive/enclosure/cable/port (although the fact that it errors on the same file every time points to the drive itself).

Last edited by Slithery (2017-05-24 22:39:06)

Arch Linux

#1 2017-05-24 20:58:17

Rsync, slow, failing to copy some files locally between external disks

#2 2017-05-24 22:12:08

Re: Rsync, slow, failing to copy some files locally between external disks

#3 2017-05-24 22:25:31

Re: Rsync, slow, failing to copy some files locally between external disks

#4 2017-05-24 22:38:19

Re: Rsync, slow, failing to copy some files locally between external disks

Board footer