new install - system freeze when copying large files

morgothsauron · 2020-06-20 17:02:15

I just installed (well 3 times in 2 days) Arch on brand new computer (i7 9700k, 16Gb RAM, Intel M2 660p series) and I'm struggling with a weird problem. I use LVM on LUKS.

The installation went fine and no specific problem to get the UEFI boot loader to work (GRUB). Dual-boot with Windows works well too.

The problem starts when I try to copy libvirt images (qcow2 files). As soon as I start copying these files (cp or rsync) the system gets unresponsive for short periods of time. Basic things like commands in terminal won't execute immediately and for commands as simple as ls. It just get stuck for 10-15 seconds.

Even rsync will temporarily stop showing progress. iotop shows high CPU usage for dmcrypt_write/2. Everything comes back to normal when I stop the file copy. I notice that the rsync speed varies a lot: I can reach up to 500MB/s and may drop to 1MB/s. I copy from a SSD connected via USB3. I never had that problem copying the same qcow2 file on my old computer.

Testing with hdparm -Tt --direct /dev/nvme0n1 do show expected speed for this M2 model

[root@morgoth /]# hdparm -Tt --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT cached reads:   2790 MB in  2.00 seconds = 1394.57 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing O_DIRECT disk reads: 2814 MB in  3.00 seconds = 937.66 MB/sec
[root@morgoth /]#

So far I was not able to find anything. At first I thought it had something to do with my LUKS and BTRFS setup so I reinstalled to use LUKS + LVM. I probably missed something, but I'm unable to find the cause of this problem.

edit: format

Last edited by morgothsauron (2020-06-20 18:45:42)

morgothsauron · 2020-06-20 18:45:23

I did 2 tests and it seems to be related to the NVME.

1) No problem if I copy the qcow2 files to a regular EXT4 partition without LUKS on a second SSD (Crucial MX300) installed in the machine. I could copy all the qcow2 files (63Gb). It was slow, but the speed was stable and I did not notice any freeze.
2) Creating a empty file with dd eventually trigger the problem.

I might just try to reinstall again, but without LUKS.

Last edited by morgothsauron (2020-06-20 18:49:16)

seth · 2020-06-20 20:22:18

https://serverfault.com/questions/30102 … encryption

loqs · 2020-06-20 20:30:13

Another explanation of LUKS performance bottlenecks https://blog.cloudflare.com/speeding-up … ncryption/

morgothsauron · 2020-06-21 09:43:19

seth wrote:

https://serverfault.com/questions/30102 … encryption

I added module aesni_intel in /etc/mkinicpio.conf and rebuild the initramfs (mkinitcpio -P).

This time I copied the qcows2 images right after the installation before installing a desktop environment. I didn't notice anything special, except the transfer speed (~ 50MB/s) but maybe there are other factors here (USB3 + cheap SanDisk SSD).

It is not the same as copying files, but I ran qemu-img convert to compress my qcow2 images. During that process there was no sign of free or slow down. Top processes/threads in iotop were qemu-img, which is expected. I did see dmcrypt_write/2 once in a while but it was there for a split second.

While qemu-img was still running I created a 20Gb empty file (dd if=/dev/zero of=/var/lib/libvrit/zero bs=1M count=20000) and didn't notice any issue. I was a bit optimistic and started to create a new empty 40Gb file. At some point I did notice the problem: the system became unresponsive for at least 30 seconds. By unresponsive I mean I couldn't switch between applications and the music player just stopped playing.

I don't know if it matters but I decided to not use compression for BTRFS. This time I added to mount options for this subvolume dedicated to libvirt: nodatacow and commit=60.

Knowing about this commit interval, I wonder if the behavior I see is inherent to btrfs in general.

Last edited by morgothsauron (2020-06-21 11:58:04)

morgothsauron · 2020-06-21 13:05:17

loqs wrote:

Another explanation of LUKS performance bottlenecks https://blog.cloudflare.com/speeding-up … ncryption/

I did read the whole article. I wonder to which extent it applies to my case. Could it be that this hardware is "so fast" that the default dm-crypt cannot work efficiently ? Based on the output from rsync, I do think it should be faster. The transfer are as fast as on my old PC with an i3 2xxx and 8Gb of RAM.

seth · 2020-06-21 13:39:12

…didn't notice any issue. I was a bit optimistic and started to create a new empty 40Gb file…

Do you have a swap partition inside the LVM?
Also see https://bbs.archlinux.org/viewtopic.php … 1#p1832171

morgothsauron · 2020-06-21 13:59:28

seth wrote:

…didn't notice any issue. I was a bit optimistic and started to create a new empty 40Gb file…
Do you have a swap partition inside the LVM?
Also see https://bbs.archlinux.org/viewtopic.php … 1#p1832171

Well, in my last post I was in fact back to BTRFS (I find it easier to manage). At the moment I don't have a swap file (still trying to figure out why swapon fails).

Should I try to apply the settings listed in the linked post (/etc/sysctl.d/40-dirty.conf) ?

Right now I'm reconsidering my choice for encryption. I never had any problem on my previous PC with Fedora and Arch. I'm surprised to see the new PC "struggle" like that.

seth · 2020-06-21 14:05:51

I'd give the altered settings a try since the scenario is very similar.
Did you old PC have an SSD?

nb. that there several reports about swap inside LVM being slow.

morgothsauron · 2020-06-21 14:19:56

The old PC had a SSD a well, a Crucial MX300 that I use as a second drive now. It was SATA2 if I remember correctly. For the past month I was using Arch without problem (LVM on LUKS) on the old machine. Before that I was runningFedora with LUKS as well (partition on LUKS).

For the new PC I decided to get a 512Gb M2 drive (it wasn't that expensive). Right now I have Arch and Windows installed on it.

I will try the settings and post back if I notice anything.

morgothsauron · 2020-06-21 15:14:47

I added the following settings to /etc/sysctl.d/40-dirty.conf

vm.dirty_background_bytes = 52428800
vm.dirty_bytes = 314572800

I started the creation of an empty file with 'dd'. I did notice some slow down, but it didn't last as long as before. But not a big difference.

I decided to check the specifications details for that M2 drive in case I missed something important
https://www.intel.com/content/www/us/en … m-3d2.html

It does says that there is AES 256 bit Hardware Encryption. So, if I understand correctly, this drive already has hardware encryption and with LUKS I'm just adding an extra layer of encryption. Do I understand this correctly ?

Edit: Ok, I might not be able to use the hardware encryption. I have a MSI board (B365M) and I didn't see any options to enable such encryption (no option to set password on a drive)

Last edited by morgothsauron (2020-06-21 15:51:51)

morgothsauron · 2020-06-21 16:11:48

Ok, since this is a new computer I will first try to run it normally without encryption. This way I'll be able to see how it performs without LUKS. I need a baseline to see how big LUKS affect the performance. I most probably don't need to encrypt the whole system in the first place. Maybe something to just crypt my home directory would be more than enough to safely store important data (taxes, invoices, identity papers, etc). Maybe it's a bit overkill to go for full disk encryption when only a small part of the system needs to be protected.

morgothsauron · 2020-06-21 20:03:09

Well, it's not a lot better without LUKS. I used a simple LVM + EXT4 and I still get slowness.

I followed some recommendations from Intel
https://www.intel.com/content/dam/suppo … 02-002.pdf

I checked that the partitions are aligned and I used ext4 without discard as described.

But I don't see any speed improvement. Copying qcow2 from a SSD can still cause some part of the system to freeze. Like there will be lag when pasting 100 lines of text in vim.

So right now I have no idea what I should do next, what should be the expected performance.

morgothsauron · 2020-06-21 20:29:08

Something I didn't think about it before: Are NVME a good option for I/O intensive tasks ? Would it be better with a "regular" SSD ?

morgothsauron · 2020-06-21 21:13:02

I did one more test: create an empty 20Gb file on both the M2 NVME drive and the Crucial MX300 SSD.

In this simple test the M2 is slower

- Crucial MX300:

# dd if=/dev/zero of=./zero bs=1M count=20000 status=progress
20887633920 bytes (21 GB, 19 GiB) copied, 51 s, 409 MB/s 
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 51.3487 s, 408 MB/s

- Intel M2:

# dd if=/dev/zero of=./zero bs=1M count=20000 status=progress
20928528384 bytes (21 GB, 19 GiB) copied, 106 s, 197 MB/s
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 106.949 s, 196 MB/s
```

Last edited by morgothsauron (2020-06-21 21:13:48)

morgothsauron · 2020-06-22 10:16:13

I looks like this "issue" is related to the M2 itself: https://www.reddit.com/r/linuxquestions … _at_times/

It's a QLC drive and according to some it can be even slower than the MX300 (SATA).

For now I'll reserve the M2 NVME for the OS and "regular" files (documents). For bigger files, like QCOW2, I'll use the MX300.

morgothsauron · 2020-06-27 19:38:45

A quick update. Since this was a new PC I could get my money back on the Intel 660P NVMe. Even if it's more expensive, I got a Samsung 970 Evo Plus 1Tb. And this is exactly what I expected from a NVMe. It's fast and I don't notice any slow down. It could be me, but I did notice a little improvement with Over Provisioning (OP). With the Samsung NVMe everything runs fine with BTRFS on a LUKS encrypted volume.

Arch Linux

#1 2020-06-20 17:02:15

new install - system freeze when copying large files

#2 2020-06-20 18:45:23

Re: new install - system freeze when copying large files

#3 2020-06-20 20:22:18

Re: new install - system freeze when copying large files

#4 2020-06-20 20:30:13

Re: new install - system freeze when copying large files

#5 2020-06-21 09:43:19

Re: new install - system freeze when copying large files

#6 2020-06-21 13:05:17

Re: new install - system freeze when copying large files

#7 2020-06-21 13:39:12

Re: new install - system freeze when copying large files

#8 2020-06-21 13:59:28

Re: new install - system freeze when copying large files

#9 2020-06-21 14:05:51

Re: new install - system freeze when copying large files

#10 2020-06-21 14:19:56

Re: new install - system freeze when copying large files

#11 2020-06-21 15:14:47

Re: new install - system freeze when copying large files

#12 2020-06-21 16:11:48

Re: new install - system freeze when copying large files

#13 2020-06-21 20:03:09

Re: new install - system freeze when copying large files

#14 2020-06-21 20:29:08

Re: new install - system freeze when copying large files

#15 2020-06-21 21:13:02

Re: new install - system freeze when copying large files

#16 2020-06-22 10:16:13

Re: new install - system freeze when copying large files

#17 2020-06-27 19:38:45

Re: new install - system freeze when copying large files

Board footer