You are not logged in.

Pages: 1

#1 2025-04-15 02:34:26

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

cp from internal SSD via USB 3 to external HDD is very slow

Hi,

I am copying from an internal SATA SSD via USB 3 to an external SATA WD Blue, a 3.5" CMR HDD.

The first part “cp -ai /mnt/winos7/” (see [*1]) took about 70 minutes for 188 GiB, almost all files are quite small. The second part “cp -ai /mnt/winos10/” (see [*1]) should copy about 100 GiB small files + about 740 GiB for a single vbox vdi file. The about 100 GiB small files were copied and about 310 GiB of the vdi file were also already copied, at 03:30, so for about 410 GiB it took about 9 hours. I think this is taking far too long. Does anyone have an idea?

FWIW ^Z was an oversight. I meant to press Ctrl+Alt+T to open a terminal tab, and apparently pressed Ctrl+Z (German QWERTZ keyboard).

CPU Model: 6.191.5 "13th Gen Intel(R) Core(TM) i3-13100"
Memory Size: 32 GB
For top see [*2].

• rocketmouse@archlinux ~
$ uname -r; cat /proc/cmdline
6.14.2-arch1-1
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0

Regards,
Ralf

PS: There are no emails for 2025 in the Arch-general mailing list archive. Was and is there no more traffic or is the list down? I sent this request to the list, it didn't came through.

[*1]

• root@archlinux /home/rocketmouse
# date; cp -ai /mnt/winos7/{*txt,o10_share,winOS_2} /mnt/ua.fantec/w7_ipad_archive-winos7_not_included-2025-apr-14/winos7/; echo $?; date; cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/; echo $?; date
Mon 14 Apr 17:02:20 CEST 2025
0
Mon 14 Apr 18:13:45 CEST 2025
^Z
[1]+  Stopped                 cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/
148
Tue 15 Apr 02:56:03 CEST 2025
• root@archlinux /home/rocketmouse
# fg 1
cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/

[*2]

top - 03:38:10 up 2 days, 17:21,  1 user,  load average: 1.30, 1.26, 1.30
Tasks: 284 total, 1 running, 281 sleep, 2 d-sleep, 0 stopped, 0 zombie
%Cpu(s):  1.8 us,  0.8 sy,  0.0 ni, 84.3 id, 12.9 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem :  31862.0 total,    316.4 free,   3401.6 used,  29130.6 buff/cache    
MiB Swap:  16383.0 total,  16382.5 free,      0.5 used.  28460.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                
  82310 rocketm+  20   0  557740  60108  48756 S  10.3   0.2   0:18.00 roxterm                                                                                                                
    663 root      20   0  819996 107492  66704 S   7.0   0.3  17:40.51 Xorg                                                                                                                   
 100095 rocketm+  20   0 2587672 172912 117660 S   1.3   0.5   0:08.42 Isolated Web Co                                                                                                        
  87583 root      20   0    7440   6136   5516 D   1.0   0.0  14:05.38 cp                                                                                                                     
   2078 rocketm+  20   0  167504  41148  33812 S   0.3   0.1   4:47.97 parcellite                                                                                                             
   2697 rocketm+  20   0   73.9g 806944 168776 S   0.3   2.5  11:00.17 evolution                                                                                                              
  96948 root      20   0       0      0      0 I   0.3   0.0   0:12.97 kworker/u32:4-events_unbound                                                                                           
 100138 root      20   0       0      0      0 I   0.3   0.0   0:00.99 kworker/1:2-events                                                                                                     
 100195 rocketm+  20   0   70.5g 163948 142808 S   0.3   0.5   0:00.67 WebKitWebProces                                                                                                        
 100326 root      20   0       0      0      0 I   0.3   0.0   0:01.75 kworker/u32:1-flush-8:80                                                                                               
 100948 rocketm+  20   0   12372   9836   7660 R   0.3   0.0   0:00.07 top                                                                                                                    
      1 root      20   0   24012  15008   9320 S   0.0   0.0   0:26.64 systemd                                                                                                                
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.23 kthreadd                                                                                                               
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqu

Last edited by Ralf (2025-04-15 02:36:09)

Offline

#2 2025-04-15 09:47:57

ReDress
Member
From: Nairobi
Registered: 2024-11-30
Posts: 133

Re: cp from internal SSD via USB 3 to external HDD is very slow

This might be a lame suggestion but for large files, 'dd' might be better choice.

dd if=infile.vdi of=outfile.vdi bs=64K conv=noerror,sync status=progress

Unlike `cp`, it allows you control the amount of bytes read/written with each cycle. Setting an optimum size here would likely greatly improve the total time taken when copying.

Offline

#3 2025-04-15 10:35:17

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

Thank you for the suggestion. At next I need to copy a Windows 11 vdi about 850 TiB. At the moment I'm doing a lot of backups and I noticed that speed goes up and down and up again. Right after posting my request the last hundred GiB were copied pretty fast. If possible I'll SMART check the disks soon (the SSDs aren't in the smartctl data base), but I suspect that the culprit isn't a drive. I'm using different source and destination drives and will see what happens after restarting the machine, before doing the next backup. I wonder what an "optimum size" for use with dd is.

Offline

#4 2025-04-15 12:16:59

ReDress
Member
From: Nairobi
Registered: 2024-11-30
Posts: 133

Re: cp from internal SSD via USB 3 to external HDD is very slow

Ralf wrote:

I wonder what an "optimum size" for use with dd is.

Now I just tested this and it seems `cp` is giving me better speeds than the best I can get with 'dd'. Guess your issue is elsewhere?

Offline

#5 2025-04-15 13:25:39

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 344

Re: cp from internal SSD via USB 3 to external HDD is very slow

Ralf wrote:

Does anyone have an idea?

What types of filesystem are mounted at /mnt/winos7 and /mnt/winos10 ?

Offline

#6 2025-04-15 13:41:36

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 344

Re: cp from internal SSD via USB 3 to external HDD is very slow

Ralf wrote:

took about 70 minutes for 188 GiB

I.e. about 46 MiB/s. It looks like USB 2.0 High Speed, not 3.0.

Offline

#7 2025-04-15 15:18:44

seth
Member
Registered: 2012-09-03
Posts: 62,803

Re: cp from internal SSD via USB 3 to external HDD is very slow

for about 410 GiB it took about 9 hours

~13MB/s (the "i" being implied…)

Do you customize https://lonesysadmin.net/2013/12/22/bet … rty_ratio/ ?
@Redress: wrt above, did you use sync cp and dd for that test?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#8 2025-04-16 00:32:25

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

All used file systems are ext4.

• root@archlinux /home/rocketmouse 
# sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200

Before I've done the next backup, I shut down the machine, restarted the machine and diff the already done backup. The diff was also terribly slow. After that I've done another backup, this time with rsync instead of cp, from an internal NVMe SSD via USB 3 to another SATA HDD, a WD Purple 5400 RPM, 3.5", CMR (the SATA HDD used before is a WD Red, 5400 RPM, 3,5", CMR, but not a WD Blue) in another enclosure, but the enclosure is the same model from the same brand with the same firmware. The PC USB port used, the USB cable and the power supply for the enclosure are exactly the same as before. This backup was fast.

• root@archlinux /home/rocketmouse 
# date; rsync -ah --xattrs --progress /mnt/winos11/winos11/ /mnt/ub.fantec/winos11-2025-apr-15/; echo $?; date 
Tue 15 Apr 20:51:54 CEST 2025
sending incremental file list
./
winos11-1.15-linux.vbox
          6.01K 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=11/13)
winos11-1.18-linux.vbox
          6.00K 100%    5.72MB/s    0:00:00 (xfr#2, to-chk=10/13)
winos11.nvram
        547.33K 100%  260.99MB/s    0:00:00 (xfr#3, to-chk=9/13)
winos11.vbox
          6.36K 100%    3.03MB/s    0:00:00 (xfr#4, to-chk=8/13)
winos11.vbox-prev
          7.52K 100%    2.39MB/s    0:00:00 (xfr#5, to-chk=7/13)
winos11.vdi
        900.01G 100%  167.32MB/s    1:25:29 (xfr#6, to-chk=6/13)
Logs/
Logs/VBox.log
        238.34K 100%  518.38kB/s    0:00:00 (xfr#7, to-chk=3/13)
Logs/VBox.log.1
        239.65K 100%  520.06kB/s    0:00:00 (xfr#8, to-chk=2/13)
Logs/VBox.log.2
        237.15K 100%  513.51kB/s    0:00:00 (xfr#9, to-chk=1/13)
Logs/VBox.log.3
        243.93K 100%  516.74kB/s    0:00:00 (xfr#10, to-chk=0/13)
Snapshots/
0
Tue 15 Apr 22:17:24 CEST 2025

After shutdown and restarting the machine a diff run fast, too.

• root@archlinux /home/rocketmouse 
# date; diff -r --no-dereference /mnt/winos11/winos11/ /mnt/ub.fantec/winos11-2025-apr-15/; echo $?; date 
Tue 15 Apr 23:15:44 CEST 2025
0
Wed 16 Apr 00:40:07 CEST 2025

I have a guess. The source drive of the first backup is likely connected to a PCIe-SATA 3.0 card (I don't want to open the PC now, but according to handwritten notes, this seems to be the case), IOW it is likely not connected to one of the mobo's SATA ports. Maybe this PCIe-SATA converter is the culprit. I suspect that cp vs rsync and NVMe SSD vs SATA SSD and a different WD Model SATA HDD aren't alone the cause for this big difference.

Last edited by Ralf (2025-04-16 00:38:23)

Offline

#9 2025-04-16 05:43:08

ReDress
Member
From: Nairobi
Registered: 2024-11-30
Posts: 133

Re: cp from internal SSD via USB 3 to external HDD is very slow

seth wrote:

Do you customize https://lonesysadmin.net/2013/12/22/bet … rty_ratio/ ?
@Redress: wrt above, did you use sync cp and dd for that test?

Not initially. Did it this time round, `cp` seems to still be doing quite good, though `dd` with abnormally large sizes of BS seem be faster.

redress@metal:~> cat /dev/random > file.bin
^C
redress@metal:~> ls -lh file.bin
-rw-r--r-- 1 redress users 7.7G Apr 16 08:09 file.bin
redress@metal:~> dd if=file.bin of=outfile2.bin bs=4M conv=noerror,sync status=progress
8145338368 bytes (8.1 GB, 7.6 GiB) copied, 31 s, 263 MB/s
1963+1 records in
1964+0 records out
8237613056 bytes (8.2 GB, 7.7 GiB) copied, 31.4373 s, 262 MB/s
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile64.bin bs=64K conv=noerror,sync status=progress
8224833536 bytes (8.2 GB, 7.7 GiB) copied, 120 s, 68.5 MB/s
125674+0 records in
125674+0 records out
8236171264 bytes (8.2 GB, 7.7 GiB) copied, 120.393 s, 68.4 MB/s
redress@metal:~> dd if=file.bin of=outfile4k.bin bs=4K conv=noerror,sync status=progress
1328979968 bytes (1.3 GB, 1.2 GiB) copied, 9 s, 148 MB/s^C
330170+0 records in
330170+0 records out
1352376320 bytes (1.4 GB, 1.3 GiB) copied, 9.34282 s, 145 MB/s
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile4k.bin bs=4K conv=noerror,sync status=progress
8225304576 bytes (8.2 GB, 7.7 GiB) copied, 170 s, 48.4 MB/s
2010784+0 records in
2010784+0 records out
8236171264 bytes (8.2 GB, 7.7 GiB) copied, 170.218 s, 48.4 MB/s
redress@metal:~> sync
redress@metal:~> cp file.bin cpfile.bin
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile8m.bin bs=8M conv=noerror,sync status=progress
8153726976 bytes (8.2 GB, 7.6 GiB) copied, 20 s, 408 MB/s
981+1 records in
982+0 records out
8237613056 bytes (8.2 GB, 7.7 GiB) copied, 20.1917 s, 408 MB/s
redress@metal:~>

Timed manually to 30 seconds

redress@metal:~> cp file.bin cpfile.bin

Offline

#10 2025-04-16 09:22:51

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

It is said that copying many small files takes longer than copying a few large files. I suspect that no one is thinking of files close to 1 TiB in size.

• rocketmouse@archlinux ~ 
$ /usr/bin/ls -hl /mnt/winos1?/winos1?/*vdi
-rw------- 1 rocketmouse rocketmouse 737G Apr 14 01:24 /mnt/winos10/winos10/winos10.vdi
-rw------- 1 rocketmouse rocketmouse 839G Apr 14 03:23 /mnt/winos11/winos11/winos11.vdi

Why does a copy take about half a day on one occasion and just under 1 1/2 hours on another?

Offline

#11 2025-04-16 09:31:49

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

ReDress wrote:

Timed manually to 30 seconds

While I'm in favour of  "date; command; date" other are used to "time".

• rocketmouse@archlinux ~ 
$ tldr time

  time

  Measure how long a command took to run.
  Note: `time` can either exist as a shell builtin, a standalone program or both.
  More information: https://manned.org/time.

  - Run the `command` and print the time measurements to `stdout`:
    time command

  - Create a very simple stopwatch (only works in Bash):
    time read

Found 2 pages with the same name under the platforms: windows, windows.

Offline

#12 2025-04-16 14:06:36

seth
Member
Registered: 2012-09-03
Posts: 62,803

Re: cp from internal SSD via USB 3 to external HDD is very slow

Why does a copy take about half a day on one occasion and just under 1 1/2 hours on another?

For clarification: same source and destination media and you're talking about the two ~800GB files?

* sideload, some indexer etc?
* media defects (IO errors)?
* thrashing? (zsawp? transparent hugepages?)

Can you measure (significantly) different performance between copying 16GB and 160GB?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#13 2025-04-16 14:34:51

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

seth wrote:

For clarification: same source and destination media and you're talking about the two ~800GB files?

Different source and destination media, but indeed those ~800GB files.

seth wrote:

* sideload, some indexer etc?

No.

seth wrote:

* media defects (IO errors)?

When running the cp command no IO errors were reported.
The destination HDDs aren't very old and I should be able to check those with smartctl. The source SSD used when copying was slow might or might not be very old, but smartctl unlikely does help to check it and the vendors software is discontinued for Linux.

seth wrote:

* thrashing? (zsawp? transparent hugepages?)

The top command didn't show a lag of memory or a CPU performance issue. Never heard of zsawp or transparent hugepages before and never enabled it myself, so no.

seth wrote:

Can you measure (significantly) different performance between copying 16GB and 160GB?

I'll have to test it after Easter, as soon as I have more time again.

Last edited by Ralf (2025-04-16 14:37:31)

Offline

#14 2025-04-16 15:23:23

seth
Member
Registered: 2012-09-03
Posts: 62,803

Re: cp from internal SSD via USB 3 to external HDD is very slow

Different source and destination media, but indeed those ~800GB files.

Take the source out of the equation and dd if=/dev/urandom or /dev/zero

Never heard of zsawp or transparent hugepages before and never enabled it myself, so no.

"Hah"
https://wiki.archlinux.org/title/Zswap (was enabled by default for a while, check the local state)

cat /sys/kernel/mm/transparent_hugepage/enabled

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#15 2025-04-16 16:02:40

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

seth wrote:

Take the source out of the equation and dd if=/dev/urandom or /dev/zero

That's a good idea for a first test.

https://wiki.archlinux.org/title/Zswap (was enabled by default for a while, check the local state)

cat /sys/kernel/mm/transparent_hugepage/enabled
• rocketmouse@archlinux ~ 
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
• rocketmouse@archlinux ~ 
$ zgrep CONFIG_ZSWAP_DEFAULT_ON /proc/config.gz
CONFIG_ZSWAP_DEFAULT_ON=y
• rocketmouse@archlinux ~ 
$ uname -a
Linux archlinux 6.14.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 18:43:59 +0000 x86_64 GNU/Linux

Offline

#16 2025-04-16 16:07:14

seth
Member
Registered: 2012-09-03
Posts: 62,803

Re: cp from internal SSD via USB 3 to external HDD is very slow

Try to disable zswap and add "transparent_hugepage=never" to the kernel parameters (you can also write "never" into that file to alter it at runtime)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#17 2025-04-16 18:10:51

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

Befor:

• rocketmouse@archlinux ~ 
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0
• rocketmouse@archlinux ~ 
$ grep -r . /sys/module/zswap/parameters/ 
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/shrinker_enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:zstd
/sys/module/zswap/parameters/zpool:zsmalloc
/sys/module/zswap/parameters/accept_threshold_percent:90
• rocketmouse@archlinux ~ 
$ cat /sys/kernel/mm/transparent_hugepage/enabled 
[always] madvise never
• rocketmouse@archlinux ~ 
$ grep zswap /boot/grub/grub.cfg -A2
menuentry "    Arch Linux no zswap, no transparent hugepages" {
        search --no-floppy --set=root --label m1.archlinux
        linux /boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0 zswap.enabled=0 transparent_hugepage=never 
        initrd /boot/intel-ucode.img /boot/initramfs-linux.img
}
• rocketmouse@archlinux ~ 
$ sudo shutdown -r now

After:

• rocketmouse@archlinux ~ 
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0 zswap.enabled=0 transparent_hugepage=never
• rocketmouse@archlinux ~ 
$ grep -r . /sys/module/zswap/parameters/ 
/sys/module/zswap/parameters/enabled:N
/sys/module/zswap/parameters/shrinker_enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:zstd
/sys/module/zswap/parameters/zpool:zsmalloc
/sys/module/zswap/parameters/accept_threshold_percent:90
• rocketmouse@archlinux ~ 
$ cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]

I don't have time for copy tests for now. But at least I can see whether the whole thing has an effect on other tasks.
Thank you for the hints.

Offline

#18 2025-04-16 19:11:44

seth
Member
Registered: 2012-09-03
Posts: 62,803

Re: cp from internal SSD via USB 3 to external HDD is very slow

"ibt=off" is most likely a residual from when the nvidia driver (or another OOT module?) didn't support that?
In that case you can probably remove that again.

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#19 2025-04-16 19:59:21

Ralf
Member
From: Ruhrgebiet
Registered: 2013-03-15
Posts: 65

Re: cp from internal SSD via USB 3 to external HDD is very slow

You are right, it was removed from Arch's VirtualBox Wiki 2 years ago.

Offline

Pages: 1

Board footer

Powered by FluxBB