You are not logged in.
- Topics: Active | Unanswered
#1 2010-11-18 06:07:40
- benoror
- Member
- Registered: 2009-07-20
- Posts: 66
- Website
Copy many big multimedia files from one partition to another
I want to copy 300~GB of data (mostly big files) from an Ext3 partition to a NTFS one, both in the same harddrive.
I have tried many alternatives, such as: cp, dd, tar, but all of them are so sick slow.
HDD: 1.5TB Seagate USB2.0
Any idea to improve the performance?
Last edited by benoror (2010-11-18 06:09:11)
ArchLinux :: Awesome WM :: dotfiles
Offline
#2 2010-11-18 06:28:28
- karol
- Archivist
- Registered: 2009-05-06
- Posts: 25,440
Re: Copy many big multimedia files from one partition to another
How fast is "sick slow"?
Do you have the drive in an external enclosure? Can you mount it via SATA instead USB 2.0?
Offline
#4 2010-11-18 07:18:58
- AugustePop
- Member
- Registered: 2010-04-27
- Posts: 95
Re: Copy many big multimedia files from one partition to another
The source and destination are on different partitions, I guess the data traveled from source to memory and then to the destination.
However, 4% of 300G in 2 hours means 1.7M/sec. It is quite slow indeed.
I am not sure if it is caused by the simultaneous read and write you did on the disk. If your system has some decently large memory, try to copy file to /dev/shm and write back to the disk one by one.
Or maybe you are simply using a legacy USB 1.0 port?
Offline
#5 2010-11-18 07:20:05
- sugardeath
- Member
- Registered: 2010-03-02
- Posts: 82
Re: Copy many big multimedia files from one partition to another
External, USB only. 4% in about 2 hours.
The operation is running on the same device, so it doesn't matter if its USB or SATA, right?
I think it would, since the data from the drive still has to travel through the CPU. SATA would get the data to the CPU faster than USB would, and then it would get to the new partition from the CPU faster as well.
I would recommend rsync. You probably won't see much improvement, but at least with rsync you can cancel it in the middle and it won't have to re-copy all of the already-copied files. DD doesn't care about files and cp will re-copy everything unless you specify exclusions (unsure about tar).
Offline
#6 2010-11-18 16:09:29
- benoror
- Member
- Registered: 2009-07-20
- Posts: 66
- Website
Re: Copy many big multimedia files from one partition to another
Thanks for the suggestions, actually tar copied everything after 12 hours, but with 8gb of RAM it would have been a good choice using /dev/shm.
ArchLinux :: Awesome WM :: dotfiles
Offline
#7 2010-11-18 20:14:43
- B-Con
- Member
- From: USA
- Registered: 2007-12-17
- Posts: 554
- Website
Re: Copy many big multimedia files from one partition to another
Was the NTFS partition mounted with -o sync? I've seen that absolutely kill my transfer rates over USB to FAT and NTFS.
[edit]
And I'm including the sync time after the asynchronous copy in the above statement. Like copy of a file with -o sync takes 30 minutes and without -o sync takes 3 minutes plus a 3 minute sync. Ridiculous difference.
Last edited by B-Con (2010-11-18 20:16:23)
Offline