Arch Linux

lapsio

Member

From: Warsaw

Registered: 2015-09-30

Posts: 50

Is there any more "official" multithread cp-like utility?

When you type "multithread cp" you'll find multiple SE threads from around 2010 or 2014 where people say it's unnecessary since cp is mostly i/o bound. However it's not 2010 anymore and Gen5 NVME ssd's in RAID0 or RAID1 are a thing and cp actually really DOES bottleneck on cpu now. I found few dodgy academic projects implemented as dirty hack patches over some very specific coreutls version but since I doubt I'm the only person in the world who wants to copy files faster I believe there probably is some more... supported?... solution to modern file copying.

I'm not talking about copying multiple small files since this use case can easily be covered by using GNU `parallel` utility and normal cp, I'm more concerned about single big files like 300 GB VM and backup images that cap at around 3 GB/s on Threadripper 5000 platform, hogging single core to 100%, despite using multiple Gen4 ssds in RAID (not to mention single such ssd easily achieves 7GB/s when using 3 threads). This issue will most likely be even more apparent on TR 7000 platform with pci-e Gen5 SSDs.

Last edited by lapsio (2024-01-28 22:28:49)

dogknowsnx

Guest

Re: Is there any more "official" multithread cp-like utility?

Not sure about the "official" part, but have you experimented with 'rsync' ?
Maybe also try picking a different I/O scheduler better fitting your needs:
https://wiki.archlinux.org/title/Improv … schedulers

willemw

Member

Registered: 2013-02-19

Posts: 116

Re: Is there any more "official" multithread cp-like utility?

Maybe a copy program (cp, rsync) in combination with mbuffer, using pipes? Probably the copy program will still be the bottleneck.

Last edited by willemw (2024-01-29 10:14:40)