Transfer of large files freezes Arch

goldmund · 2021-02-15 18:44:01

Hi,

I have just installed Arch after my 11-year-long journey through Ubuntu and Manjaro. The installation was easy and (almost) everything is much snappier. It's all extremely well documented, thank you all for your contributions!

Now to the problem: when I am transferring large files (e.g. video files) from my external to my internal drive with rsync (or other tools), at first it's showing ridiculously high speeds (150MB/s for a HDD) which quickly slows down to some 5MB/s and partially freezes the system. I can interrupt the process, but can't open new terminal windows etc. During this time the displayed speed fluctuates, anything from 5MB/s to 150MB/s.

I have tried this with multiple drives (SSDs and HDDs, all using ext4 filesystem) and saw the same freezes, so it's not a hardware problem – moreover, I saw nothing of this behaviour with Manjaro on the same machine. To my annoyance, journalctl shows no errors.

I found a similar post on Reddit, however, none of the options mentioned changes anything.

I have tried:

lowering the amount of dirty vm
setting transparent hugepage to always, madvise or never
changing i/o scheduler to bfq

My internal drive, same as previously Manjaro, has LUKS encryption on the root partition. Here is some system info:

System:    Host: acer Kernel: 5.10.16-arch1-1 x86_64 bits: 64 Desktop: dwm 6.2 Distro: Arch Linux 
Machine:   Type: Laptop System: Acer product: Aspire A515-52 v: V1.04 serial: <superuser required> 
           Mobo: WL model: Raticate_WL v: V1.04 serial: <superuser required> UEFI: Insyde v: 1.04 
           date: 09/06/2018 
CPU:       Info: Quad Core Intel Core i5-8265U [MT MCP] speed: 600 MHz min/max: 400/3900 MHz 
Drives:    Local Storage: total: 3.87 TiB used: 1.93 TiB (49.9%) 
           ID-1: /dev/sda vendor: SanDisk model: SD9SN8W256G1014 size: 238.47 GiB 
           ID-2: /dev/sdb type: USB vendor: Seagate model: Expansion size: 3.64 TiB 
Info:      Processes: 217 Uptime: 1h 28m Memory: 7.62 GiB used: 1.78 GiB (23.3%) Shell: Zsh

Do any of you have any ideas how to solve this?

Thank you!

loqs · 2021-02-15 20:25:59

If you open top in a terminal window then start a large file transfer do you see one or more processes in the D state when the system is not responding?

goldmund · 2021-02-15 20:45:19

loqs wrote:

If you open top in a terminal window then start a large file transfer do you see one or more processes in the D state when the system is not responding?

Yes, a couple of kworker/u16:12+kcryptd/254:0 are in the D state during the transfer.

To add, it seems to be even worse when the external drive is also encrypted (with LUKS). I have tried both transfers with and without encryption.

seth · 2021-02-15 21:56:04

sysctl -a | grep dirty

Lower vm.dirty_background_ratio well below vm.dirty_ratio
Once you hit vm.dirty_background_ratio, the kworker kicks in. vm.dirty_ratio is a hard limit and if you reach it, the I/O will block until you freed some dirty pages.

You're reading from a fast source (150MB/s) to a slow (LUKS) destination, so you need to widen the gap to prevent hitting vm.dirty_ratio

goldmund · 2021-02-15 23:42:33

seth wrote:

Lower vm.dirty_background_ratio well below vm.dirty_ratio
Once you hit vm.dirty_background_ratio, the kworker kicks in. vm.dirty_ratio is a hard limit and if you reach it, the I/O will block until you freed some dirty pages.
You're reading from a fast source (150MB/s) to a slow (LUKS) destination, so you need to widen the gap to prevent hitting vm.dirty_ratio

Thanks, I have now increased the range between the two values:

vm.dirty_background_ratio = 3
vm.dirty_ratio = 15

However, it doesn't seem to make much of a difference, Arch still freezes during transfers.

I have realized that I have reorganized my USB ports before the Arch install and I am now using USB 3.0 instead of 2.0 for transfers. I should have thought of this before! This explains the fast transfer speeds and why my laptop struggles to process data from one LUKS volume to another. I guess I will go back to USB 2.0.

progandy · 2021-02-15 23:59:22

You could also try to run rsync with nice, ionice or the --bwlimit option.

seth · 2021-02-16 07:12:09

Furthermore you may be able to speed up the encryption:
"cat /proc/cpuinfo" - does your CPU support AES-NI?
Is the module loaded and did you AES encrypt the disk?

goldmund · 2021-02-16 13:29:06

progandy wrote:

You could also try to run rsync with nice, ionice or the --bwlimit option.

That's a good idea. I have tried it out and it looks promising. I have transferred a nearly 8GB video file from a LUKS external HDD to my LUKS internal SSD through USB 3.0, using various settings.

Since everyone loves benchmarking, I am posting the results here, all sorted by a "freeze index" from 0 (system completely responsive) to 10 (can't move the mouse or do anything).

# freeze index: 5
rsync -avP --bwlimit=50m video.mkv ~/
7,719,407,560 100%   22.80MB/s    0:05:22

# freeze index: 7
rsync -avP video.mkv ~/
7,719,407,560 100%   38.37MB/s    0:03:11

# freeze index: 7
nice -n 19 rsync -avP video.mkv ~/
7,719,407,560 100%   36.75MB/s    0:03:20

# freeze index: 7
nice -n 19 rsync -avP --bwlimit=50m video.mkv ~/
7,719,407,560 100%   29.18MB/s    0:04:12

# freeze index: 8
ionice -c 3 rsync -avP video.mkv ~/
7,719,407,560 100%   26.83MB/s    0:04:34

And a plain rsync on USB 2.0:

# freeze index: 5
rsync -avP video.mkv ~/
7,719,407,560 100%   24.97MB/s    0:04:54

Although everything is much slower during transfers, limiting speeds clearly helps increase responsiveness of the system.

goldmund · 2021-02-16 13:38:55

seth wrote:

Furthermore you may be able to speed up the encryption:
"cat /proc/cpuinfo" - does your CPU support AES-NI?
Is the module loaded and did you AES encrypt the disk?

Yes, /proc/cpuinfo seems to indicate that "aes" is available.

Another check:

$ cpuid | grep -i aes | sort | uniq
AES instruction                         = true
VAES instructions                        = false

Both my external HDD and my internal SSD are aes-xts-plain64 encrypted.

seth · 2021-02-16 14:13:54

https://bbs.archlinux.org/viewtopic.php … 6#p1839696

Move the aesni_intel module to the initramfs in doubt.

goldmund · 2021-02-16 15:06:22

seth wrote:

https://bbs.archlinux.org/viewtopic.php … 6#p1839696
Move the aesni_intel module to the initramfs in doubt.

I hope I followed your advice correctly. I have added aesni_intel to /etc/mkinitcpio.conf in MODULES, ran sudo mkinitcpio -P and rebooted. Anyhow, this unfortunately doesn't seem to change the problem with the freezing.

seth · 2021-02-16 15:19:18

Does dmesg suggest it's in use?

goldmund · 2021-02-16 15:56:51

seth wrote:

Does dmesg suggest it's in use?

Yes, it seems so:

# dmesg | grep -i aes
[    0.973521] AES CTR mode by8 optimization enabled
[   33.559577] RTW: set pairwise key camid:4, addr:2c:3a:fd:a5:d9:e9, kid:0, type:AES
[   33.560882] RTW: set group key camid:5, addr:2c:3a:fd:a5:d9:e9, kid:1, type:AES
[  211.485449] RTW: set group key camid:6, addr:2c:3a:fd:a5:d9:e9, kid:2, type:AES

seth · 2021-02-16 16:01:42

dmesg | grep crypt

goldmund · 2021-02-16 16:47:02

# dmesg | grep crypt
[    0.000000] Command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot root=/dev/mapper/cryptroot rw net.ifnames=0
[    0.040573] Kernel command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot root=/dev/mapper/cryptroot rw net.ifnames=0
[    0.505136] Key type ._fscrypt registered
[    0.505137] Key type .fscrypt registered
[    0.505137] Key type fscrypt-provisioning registered
[    0.790017] Freeing unused decrypted memory: 2036K
[    0.854883]     cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot
[    0.970558] cryptd: max_cpu_qlen set to 1000
[    1.079971] Key type encrypted registered
[    1.110652] random: cryptsetup: uninitialized urandom read (4 bytes read)
[    1.118575] random: cryptsetup: uninitialized urandom read (4 bytes read)

seth · 2021-02-16 21:11:29

Ah, the used implementation is only printed at loglevel 7.

goldmund · 2021-02-17 09:53:11

seth wrote:

Ah, the used implementation is only printed at loglevel 7.

I've just tested it with loglevel 7 and the output of "dmesg | grep crypt" is exactly the same as above.

Do you suppose the module somehow doesn't get loaded?

seth · 2021-02-17 13:03:23

You're supposed to get *something* along "device-mapper: crypt: xts(aes) using implementation" regardless of the implementation in use (see the linked thread)

How did you set the loglevel? kernel commandline?
What if you pass "ignore_loglevel" instead?

Edit: maybe also see

cryptsetup benchmark

on what you can reasonably expect.

Last edited by seth (2021-02-17 13:04:35)

goldmund · 2021-02-17 17:15:08

Thanks, I passed "ignore_loglevel" into the kernel command line and now see this:

# dmesg | grep -i crypt
[    0.000000] Command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot root=/dev/mapper/cryptroot rw net.ifnames=0 ignore_loglevel
[    0.041050] Kernel command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot root=/dev/mapper/cryptroot rw net.ifnames=0 ignore_loglevel
[    0.535527] Key type ._fscrypt registered
[    0.536421] Key type .fscrypt registered
[    0.537302] Key type fscrypt-provisioning registered
[    0.822733] Freeing unused decrypted memory: 2036K
[    0.890461]     cryptdevice=PARTUUID=33510605-5fcc-274a-b778-c92e380462a8:cryptroot
[    0.997584] cryptd: max_cpu_qlen set to 1000
[    1.109014] Key type encrypted registered
[    1.134923] random: cryptsetup: uninitialized urandom read (4 bytes read)
[    1.142825] random: cryptsetup: uninitialized urandom read (4 bytes read)
[   44.564889] systemd[1]: systemd 247.3-1-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[   44.890649] systemd[1]: Created slice Cryptsetup Units Slice.

This might also be of interest:

# dmesg | grep -i aes
[    1.002193] AES CTR mode by8 optimization enabled
[   57.126836] RTW: set pairwise key camid:4, addr:2c:3a:fd:a5:d9:e9, kid:0, type:AES
[   57.134717] RTW: set group key camid:5, addr:2c:3a:fd:a5:d9:e9, kid:1, type:AES
[  153.319073] RTW: set group key camid:6, addr:2c:3a:fd:a5:d9:e9, kid:2, type:AES

goldmund · 2021-02-17 17:17:55

This seems fast enough:

$ cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1738932 iterations per second for 256-bit key
PBKDF2-sha256    2193673 iterations per second for 256-bit key
PBKDF2-sha512    1572077 iterations per second for 256-bit key
PBKDF2-ripemd160  893165 iterations per second for 256-bit key
PBKDF2-whirlpool  672164 iterations per second for 256-bit key
argon2i       7 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      7 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b      1159.9 MiB/s      3427.1 MiB/s
    serpent-cbc        128b        96.6 MiB/s       740.8 MiB/s
    twofish-cbc        128b       213.1 MiB/s       392.6 MiB/s
        aes-cbc        256b       883.0 MiB/s      2769.5 MiB/s
    serpent-cbc        256b        98.5 MiB/s       741.3 MiB/s
    twofish-cbc        256b       217.3 MiB/s       392.7 MiB/s
        aes-xts        256b      2021.8 MiB/s      2078.2 MiB/s
    serpent-xts        256b       716.0 MiB/s       705.1 MiB/s
    twofish-xts        256b       392.1 MiB/s       392.5 MiB/s
        aes-xts        512b      1829.4 MiB/s      1888.3 MiB/s
    serpent-xts        512b       719.0 MiB/s       706.0 MiB/s
    twofish-xts        512b       392.8 MiB/s       393.3 MiB/s

Arch Linux

#1 2021-02-15 18:44:01

Transfer of large files freezes Arch

#2 2021-02-15 20:25:59

Re: Transfer of large files freezes Arch

#3 2021-02-15 20:45:19

Re: Transfer of large files freezes Arch

#4 2021-02-15 21:56:04

Re: Transfer of large files freezes Arch

#5 2021-02-15 23:42:33

Re: Transfer of large files freezes Arch

#6 2021-02-15 23:59:22

Re: Transfer of large files freezes Arch

#7 2021-02-16 07:12:09

Re: Transfer of large files freezes Arch

#8 2021-02-16 13:29:06

Re: Transfer of large files freezes Arch

#9 2021-02-16 13:38:55

Re: Transfer of large files freezes Arch

#10 2021-02-16 14:13:54

Re: Transfer of large files freezes Arch

#11 2021-02-16 15:06:22

Re: Transfer of large files freezes Arch

#12 2021-02-16 15:19:18

Re: Transfer of large files freezes Arch

#13 2021-02-16 15:56:51

Re: Transfer of large files freezes Arch

#14 2021-02-16 16:01:42

Re: Transfer of large files freezes Arch

#15 2021-02-16 16:47:02

Re: Transfer of large files freezes Arch

#16 2021-02-16 21:11:29

Re: Transfer of large files freezes Arch

#17 2021-02-17 09:53:11

Re: Transfer of large files freezes Arch

#18 2021-02-17 13:03:23

Re: Transfer of large files freezes Arch

#19 2021-02-17 17:15:08

Re: Transfer of large files freezes Arch

#20 2021-02-17 17:17:55

Re: Transfer of large files freezes Arch

Board footer