You are not logged in.
Hi,
I am copying from an internal SATA SSD via USB 3 to an external SATA WD Blue, a 3.5" CMR HDD.
The first part “cp -ai /mnt/winos7/” (see [*1]) took about 70 minutes for 188 GiB, almost all files are quite small. The second part “cp -ai /mnt/winos10/” (see [*1]) should copy about 100 GiB small files + about 740 GiB for a single vbox vdi file. The about 100 GiB small files were copied and about 310 GiB of the vdi file were also already copied, at 03:30, so for about 410 GiB it took about 9 hours. I think this is taking far too long. Does anyone have an idea?
FWIW ^Z was an oversight. I meant to press Ctrl+Alt+T to open a terminal tab, and apparently pressed Ctrl+Z (German QWERTZ keyboard).
CPU Model: 6.191.5 "13th Gen Intel(R) Core(TM) i3-13100"
Memory Size: 32 GB
For top see [*2].
• rocketmouse@archlinux ~
$ uname -r; cat /proc/cmdline
6.14.2-arch1-1
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0
Regards,
Ralf
PS: There are no emails for 2025 in the Arch-general mailing list archive. Was and is there no more traffic or is the list down? I sent this request to the list, it didn't came through.
[*1]
• root@archlinux /home/rocketmouse
# date; cp -ai /mnt/winos7/{*txt,o10_share,winOS_2} /mnt/ua.fantec/w7_ipad_archive-winos7_not_included-2025-apr-14/winos7/; echo $?; date; cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/; echo $?; date
Mon 14 Apr 17:02:20 CEST 2025
0
Mon 14 Apr 18:13:45 CEST 2025
^Z
[1]+ Stopped cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/
148
Tue 15 Apr 02:56:03 CEST 2025
• root@archlinux /home/rocketmouse
# fg 1
cp -ai /mnt/winos10/ /mnt/ua.fantec/w10ipad_archive+winos10_is_included-2025-apr-14/
[*2]
top - 03:38:10 up 2 days, 17:21, 1 user, load average: 1.30, 1.26, 1.30
Tasks: 284 total, 1 running, 281 sleep, 2 d-sleep, 0 stopped, 0 zombie
%Cpu(s): 1.8 us, 0.8 sy, 0.0 ni, 84.3 id, 12.9 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 31862.0 total, 316.4 free, 3401.6 used, 29130.6 buff/cache
MiB Swap: 16383.0 total, 16382.5 free, 0.5 used. 28460.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
82310 rocketm+ 20 0 557740 60108 48756 S 10.3 0.2 0:18.00 roxterm
663 root 20 0 819996 107492 66704 S 7.0 0.3 17:40.51 Xorg
100095 rocketm+ 20 0 2587672 172912 117660 S 1.3 0.5 0:08.42 Isolated Web Co
87583 root 20 0 7440 6136 5516 D 1.0 0.0 14:05.38 cp
2078 rocketm+ 20 0 167504 41148 33812 S 0.3 0.1 4:47.97 parcellite
2697 rocketm+ 20 0 73.9g 806944 168776 S 0.3 2.5 11:00.17 evolution
96948 root 20 0 0 0 0 I 0.3 0.0 0:12.97 kworker/u32:4-events_unbound
100138 root 20 0 0 0 0 I 0.3 0.0 0:00.99 kworker/1:2-events
100195 rocketm+ 20 0 70.5g 163948 142808 S 0.3 0.5 0:00.67 WebKitWebProces
100326 root 20 0 0 0 0 I 0.3 0.0 0:01.75 kworker/u32:1-flush-8:80
100948 rocketm+ 20 0 12372 9836 7660 R 0.3 0.0 0:00.07 top
1 root 20 0 24012 15008 9320 S 0.0 0.0 0:26.64 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.23 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_workqu
Last edited by Ralf (2025-04-15 02:36:09)
Offline
This might be a lame suggestion but for large files, 'dd' might be better choice.
dd if=infile.vdi of=outfile.vdi bs=64K conv=noerror,sync status=progress
Unlike `cp`, it allows you control the amount of bytes read/written with each cycle. Setting an optimum size here would likely greatly improve the total time taken when copying.
Offline
Thank you for the suggestion. At next I need to copy a Windows 11 vdi about 850 TiB. At the moment I'm doing a lot of backups and I noticed that speed goes up and down and up again. Right after posting my request the last hundred GiB were copied pretty fast. If possible I'll SMART check the disks soon (the SSDs aren't in the smartctl data base), but I suspect that the culprit isn't a drive. I'm using different source and destination drives and will see what happens after restarting the machine, before doing the next backup. I wonder what an "optimum size" for use with dd is.
Offline
I wonder what an "optimum size" for use with dd is.
Now I just tested this and it seems `cp` is giving me better speeds than the best I can get with 'dd'. Guess your issue is elsewhere?
Offline
Does anyone have an idea?
What types of filesystem are mounted at /mnt/winos7 and /mnt/winos10 ?
Offline
took about 70 minutes for 188 GiB
I.e. about 46 MiB/s. It looks like USB 2.0 High Speed, not 3.0.
Offline
for about 410 GiB it took about 9 hours
~13MB/s (the "i" being implied…)
Do you customize https://lonesysadmin.net/2013/12/22/bet … rty_ratio/ ?
@Redress: wrt above, did you use sync cp and dd for that test?
Online
All used file systems are ext4.
• root@archlinux /home/rocketmouse
# sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
Before I've done the next backup, I shut down the machine, restarted the machine and diff the already done backup. The diff was also terribly slow. After that I've done another backup, this time with rsync instead of cp, from an internal NVMe SSD via USB 3 to another SATA HDD, a WD Purple 5400 RPM, 3.5", CMR (the SATA HDD used before is a WD Red, 5400 RPM, 3,5", CMR, but not a WD Blue) in another enclosure, but the enclosure is the same model from the same brand with the same firmware. The PC USB port used, the USB cable and the power supply for the enclosure are exactly the same as before. This backup was fast.
• root@archlinux /home/rocketmouse
# date; rsync -ah --xattrs --progress /mnt/winos11/winos11/ /mnt/ub.fantec/winos11-2025-apr-15/; echo $?; date
Tue 15 Apr 20:51:54 CEST 2025
sending incremental file list
./
winos11-1.15-linux.vbox
6.01K 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=11/13)
winos11-1.18-linux.vbox
6.00K 100% 5.72MB/s 0:00:00 (xfr#2, to-chk=10/13)
winos11.nvram
547.33K 100% 260.99MB/s 0:00:00 (xfr#3, to-chk=9/13)
winos11.vbox
6.36K 100% 3.03MB/s 0:00:00 (xfr#4, to-chk=8/13)
winos11.vbox-prev
7.52K 100% 2.39MB/s 0:00:00 (xfr#5, to-chk=7/13)
winos11.vdi
900.01G 100% 167.32MB/s 1:25:29 (xfr#6, to-chk=6/13)
Logs/
Logs/VBox.log
238.34K 100% 518.38kB/s 0:00:00 (xfr#7, to-chk=3/13)
Logs/VBox.log.1
239.65K 100% 520.06kB/s 0:00:00 (xfr#8, to-chk=2/13)
Logs/VBox.log.2
237.15K 100% 513.51kB/s 0:00:00 (xfr#9, to-chk=1/13)
Logs/VBox.log.3
243.93K 100% 516.74kB/s 0:00:00 (xfr#10, to-chk=0/13)
Snapshots/
0
Tue 15 Apr 22:17:24 CEST 2025
After shutdown and restarting the machine a diff run fast, too.
• root@archlinux /home/rocketmouse
# date; diff -r --no-dereference /mnt/winos11/winos11/ /mnt/ub.fantec/winos11-2025-apr-15/; echo $?; date
Tue 15 Apr 23:15:44 CEST 2025
0
Wed 16 Apr 00:40:07 CEST 2025
I have a guess. The source drive of the first backup is likely connected to a PCIe-SATA 3.0 card (I don't want to open the PC now, but according to handwritten notes, this seems to be the case), IOW it is likely not connected to one of the mobo's SATA ports. Maybe this PCIe-SATA converter is the culprit. I suspect that cp vs rsync and NVMe SSD vs SATA SSD and a different WD Model SATA HDD aren't alone the cause for this big difference.
Last edited by Ralf (2025-04-16 00:38:23)
Offline
Do you customize https://lonesysadmin.net/2013/12/22/bet … rty_ratio/ ?
@Redress: wrt above, did you use sync cp and dd for that test?
Not initially. Did it this time round, `cp` seems to still be doing quite good, though `dd` with abnormally large sizes of BS seem be faster.
redress@metal:~> cat /dev/random > file.bin
^C
redress@metal:~> ls -lh file.bin
-rw-r--r-- 1 redress users 7.7G Apr 16 08:09 file.bin
redress@metal:~> dd if=file.bin of=outfile2.bin bs=4M conv=noerror,sync status=progress
8145338368 bytes (8.1 GB, 7.6 GiB) copied, 31 s, 263 MB/s
1963+1 records in
1964+0 records out
8237613056 bytes (8.2 GB, 7.7 GiB) copied, 31.4373 s, 262 MB/s
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile64.bin bs=64K conv=noerror,sync status=progress
8224833536 bytes (8.2 GB, 7.7 GiB) copied, 120 s, 68.5 MB/s
125674+0 records in
125674+0 records out
8236171264 bytes (8.2 GB, 7.7 GiB) copied, 120.393 s, 68.4 MB/s
redress@metal:~> dd if=file.bin of=outfile4k.bin bs=4K conv=noerror,sync status=progress
1328979968 bytes (1.3 GB, 1.2 GiB) copied, 9 s, 148 MB/s^C
330170+0 records in
330170+0 records out
1352376320 bytes (1.4 GB, 1.3 GiB) copied, 9.34282 s, 145 MB/s
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile4k.bin bs=4K conv=noerror,sync status=progress
8225304576 bytes (8.2 GB, 7.7 GiB) copied, 170 s, 48.4 MB/s
2010784+0 records in
2010784+0 records out
8236171264 bytes (8.2 GB, 7.7 GiB) copied, 170.218 s, 48.4 MB/s
redress@metal:~> sync
redress@metal:~> cp file.bin cpfile.bin
redress@metal:~> sync
redress@metal:~> dd if=file.bin of=outfile8m.bin bs=8M conv=noerror,sync status=progress
8153726976 bytes (8.2 GB, 7.6 GiB) copied, 20 s, 408 MB/s
981+1 records in
982+0 records out
8237613056 bytes (8.2 GB, 7.7 GiB) copied, 20.1917 s, 408 MB/s
redress@metal:~>
Timed manually to 30 seconds
redress@metal:~> cp file.bin cpfile.bin
Offline
It is said that copying many small files takes longer than copying a few large files. I suspect that no one is thinking of files close to 1 TiB in size.
• rocketmouse@archlinux ~
$ /usr/bin/ls -hl /mnt/winos1?/winos1?/*vdi
-rw------- 1 rocketmouse rocketmouse 737G Apr 14 01:24 /mnt/winos10/winos10/winos10.vdi
-rw------- 1 rocketmouse rocketmouse 839G Apr 14 03:23 /mnt/winos11/winos11/winos11.vdi
Why does a copy take about half a day on one occasion and just under 1 1/2 hours on another?
Offline
Timed manually to 30 seconds
While I'm in favour of "date; command; date" other are used to "time".
• rocketmouse@archlinux ~
$ tldr time
time
Measure how long a command took to run.
Note: `time` can either exist as a shell builtin, a standalone program or both.
More information: https://manned.org/time.
- Run the `command` and print the time measurements to `stdout`:
time command
- Create a very simple stopwatch (only works in Bash):
time read
Found 2 pages with the same name under the platforms: windows, windows.
Offline
Why does a copy take about half a day on one occasion and just under 1 1/2 hours on another?
For clarification: same source and destination media and you're talking about the two ~800GB files?
* sideload, some indexer etc?
* media defects (IO errors)?
* thrashing? (zsawp? transparent hugepages?)
Can you measure (significantly) different performance between copying 16GB and 160GB?
Online
For clarification: same source and destination media and you're talking about the two ~800GB files?
Different source and destination media, but indeed those ~800GB files.
* sideload, some indexer etc?
No.
* media defects (IO errors)?
When running the cp command no IO errors were reported.
The destination HDDs aren't very old and I should be able to check those with smartctl. The source SSD used when copying was slow might or might not be very old, but smartctl unlikely does help to check it and the vendors software is discontinued for Linux.
* thrashing? (zsawp? transparent hugepages?)
The top command didn't show a lag of memory or a CPU performance issue. Never heard of zsawp or transparent hugepages before and never enabled it myself, so no.
Can you measure (significantly) different performance between copying 16GB and 160GB?
I'll have to test it after Easter, as soon as I have more time again.
Last edited by Ralf (2025-04-16 14:37:31)
Offline
Different source and destination media, but indeed those ~800GB files.
Take the source out of the equation and dd if=/dev/urandom or /dev/zero
Never heard of zsawp or transparent hugepages before and never enabled it myself, so no.
"Hah"
https://wiki.archlinux.org/title/Zswap (was enabled by default for a while, check the local state)
cat /sys/kernel/mm/transparent_hugepage/enabled
Online
Take the source out of the equation and dd if=/dev/urandom or /dev/zero
That's a good idea for a first test.
https://wiki.archlinux.org/title/Zswap (was enabled by default for a while, check the local state)
cat /sys/kernel/mm/transparent_hugepage/enabled
• rocketmouse@archlinux ~
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
• rocketmouse@archlinux ~
$ zgrep CONFIG_ZSWAP_DEFAULT_ON /proc/config.gz
CONFIG_ZSWAP_DEFAULT_ON=y
• rocketmouse@archlinux ~
$ uname -a
Linux archlinux 6.14.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 18:43:59 +0000 x86_64 GNU/Linux
Offline
Try to disable zswap and add "transparent_hugepage=never" to the kernel parameters (you can also write "never" into that file to alter it at runtime)
Online
Befor:
• rocketmouse@archlinux ~
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0
• rocketmouse@archlinux ~
$ grep -r . /sys/module/zswap/parameters/
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/shrinker_enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:zstd
/sys/module/zswap/parameters/zpool:zsmalloc
/sys/module/zswap/parameters/accept_threshold_percent:90
• rocketmouse@archlinux ~
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
• rocketmouse@archlinux ~
$ grep zswap /boot/grub/grub.cfg -A2
menuentry " Arch Linux no zswap, no transparent hugepages" {
search --no-floppy --set=root --label m1.archlinux
linux /boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0 zswap.enabled=0 transparent_hugepage=never
initrd /boot/intel-ucode.img /boot/initramfs-linux.img
}
• rocketmouse@archlinux ~
$ sudo shutdown -r now
After:
• rocketmouse@archlinux ~
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ibt=off ipv6.disable=1 kvm.enable_virt_at_load=0 zswap.enabled=0 transparent_hugepage=never
• rocketmouse@archlinux ~
$ grep -r . /sys/module/zswap/parameters/
/sys/module/zswap/parameters/enabled:N
/sys/module/zswap/parameters/shrinker_enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:zstd
/sys/module/zswap/parameters/zpool:zsmalloc
/sys/module/zswap/parameters/accept_threshold_percent:90
• rocketmouse@archlinux ~
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
I don't have time for copy tests for now. But at least I can see whether the whole thing has an effect on other tasks.
Thank you for the hints.
Offline
"ibt=off" is most likely a residual from when the nvidia driver (or another OOT module?) didn't support that?
In that case you can probably remove that again.
Online
Offline