You are not logged in.
I was trying to determine if dm-crypt + LUKS would be suitable for my system and I dug up some simple dd-based tests. I'm aware that dd is not the best utility to be used for something like this, but my intention was only to get an estimate of relative performance. What I found alarming was when I tested the latency of the disk (but it could be that I'm interpreting these numbers wrong and this should belong in the newbie subforum).
The disk is Samsung 980 Pro 2 TB (specification says read 7000 MB/s, write 5100 MB/s).
When measuring latency I get the following:
Samsung 980 Pro NVMe (no encryption, but results similar when encrypted, see below)
# dd if=/dev/zero of=latency.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 6.53986 s, 78.3 kB/sFor comparison purposes I did the same measurement on a Kingston A400 SSD at the end of a SATAIII cable:
Kingston A400 SSD (no encryption)
# dd if=/dev/zero of=latency.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 1.66761 s, 307 kB/sAdditionally I executed the same test on another computer with a different NVMe
WD Blue SN570 1TB (different PC, also running Arch, Opal 2 encrypted drive)
# dd if=/dev/zero of=latency.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 0,511011 s, 1,0 MB/sAny idea what I'm doing wrong? Why is the performance so poor with the 980 Pro? All of the aforementioned partitions span the entire disk (max sectors) and have been formatted with mkfs.ext4 (default params).
Heres some additional tests, should they be useful:
# nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 S6WRNS0RC09710L Samsung 980 PRO with Heatsink 2TB 1 1.22 GB / 2.00 TB 512 B + 0 B 5B2QGXA7
# hdparm -Tt --direct /dev/nvme0n1
/dev/nvme0n1:
Timing O_DIRECT cached reads: 5864 MB in 2.00 seconds = 2935.94 MB/sec
Timing O_DIRECT disk reads: 12364 MB in 3.00 seconds = 4120.83 MB/sec
---- dm-crypt + LUKS (1 partition, max size) ----
# dmsetup table
root: 0 3906994176 crypt aes-xts-plain64 :64:logon:cryptsetup:8b61f498-aeab-45ec-8c7a-bfdf2edd578c-d0 0 259:1 32768 2 no_read_workqueue no_write_workqueue
# dd if=/dev/zero of=test1.img bs=1G count=1 oflag=dsync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.927435 s, 1.2 GB/s
# dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 6.53986 s, 78.3 kB/s
# echo 3 | sudo tee /proc/sys/vm/drop_caches; time dd if=big.zip of=/dev/null bs=8k
6348840967 bytes (6.3 GB, 5.9 GiB) copied, 3.18166 s, 2.0 GB/s
real 0m3.274s
---- ext4 without encryption (1 partition, max size) ----
# dd if=/dev/zero of=test1.img bs=1G count=1 oflag=dsync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.685321 s, 1.6 GB/s
# dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 6.54358 s, 78.2 kB/s
# echo 3 | sudo tee /proc/sys/vm/drop_caches; time dd if=big.zip of=/dev/null bs=8k
6348840967 bytes (6.3 GB, 5.9 GiB) copied, 2.18744 s, 2.9 GB/s
real 0m2.286sAdditional things that have not helped:
- Motherboard UEFI update
- Samsung firmware update
- Switching M2 slot
- Disabling PCI Express power saving features
- Forcing PCIE 4.0
Last edited by peabrain (2022-08-20 09:32:54)
Offline
SSD has 4k native sector sizes, with 512b emulated (for boot compatibility). When writing 512b synchronously, the SSD has to work hard re-allocating 4k sectors with each 512b write. Short story, don't do <4k writes as it'll drastically reduce the life (TBW) of your drive.
SSDs that seem fast on 512b synchronous writes are lying: they're reporting sync'd to disk when in fact just in DRAM. For more info see: I tested four NVMe SSDs from four vendors – half lose FLUSH’d data on power loss
Pro SSDs tend not to lie/cheat on benchmarks like consumer SSDs do.
Last edited by sabroad (2022-08-19 11:52:08)
--
saint_abroad
Offline
Thanks, I'll look into that information. But I still have trouble understanding some things. The nvme id-ns for this disk gives exactly the results as in the wiki:
# nvme id-ns /dev/nvme0n1
...
lbaf 0 : ms:0 lbads:9 rp:0 (in use)about which the wiki page says
It has an lbads (LBA data size) of 9, which means sectors are 2^9 or 512 bytes. If the device is capable of 4 KiB sectors, there will be another entry here with an lbads of 12.
There's only lbads:9 in my output.
Also, I tested the performance with larger block sizes also but the Samsung is till underperforming.
bs=4K count=1000 oflag=dsync
Samsung: 6.6 s | 622 kB / s
Kingston: 1.7 s | 2.4 MB / s
WD Blue: 0.5 s | 7.7 MB / s
bs=1M count=1000 oflag=dsync
Samsung: 8.7 s | 120 MB / s
Kingston: 6.3 s | 165 MB / s
WD Blue: 2.0 s | 514 MB / s
Can this also be attributed to Kingston and WD giving out false information?
Last edited by peabrain (2022-08-19 12:18:59)
Offline
With what rationale are you using dsync instead of fsync?
fsync - flush (sync) data and metadata before finishing
dsync - flush (sync) data each block
Non-enterprise SSDs can be expected to perform up to ~300 sync/s - anything more than this is lying. Why Consumer SSD Reviews are Useless for Database Performance Use Case
--
saint_abroad
Offline
My understanding is too superficial to evaluate the meaningful difference between fsync and dsync. I picked dsync up from some blog that I was reading and was baffled as to why the best device should produce the worst readings. For example:
https://www.cyberciti.biz/faq/howto-lin … d-command/
oflag=dsync (oflag=dsync) : Use synchronized I/O for data. Do not skip this option. This option get rid of caching and gives you good and accurate results
In any case, replacing oflag=dsync with conv=fsync (bs=4K, count=1000) decreases the difference but puts the Sata3 Kingston SSD above the NVMEs:
Kingston: 0.01 s | 290 MB/s
WD Blue: 0.02 s | 230 MB/s
Samsung: 0.02 s| 214 MB/s
Offline
Well, if anybody's facing the same problem, you could try switching off APST support: https://wiki.archlinux.org/title/Solid_ … ST_support
I still don't know if dd is the correct program for measuring latency. I switched to ioping (available in the community repo), and it too produced bad results:
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=2 time=4.60 ms
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=3 time=4.59 ms
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=4 time=4.59 msThese should be microseconds, not milliseconds. It seems others, too, have faced this problem: https://forum.level1techs.com/t/slow-io … ard/151970
After adding nvme_core.default_ps_max_latency_us=0 to kernel params the results are much closer to what should be expected:
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=2 time=589.0 us
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=3 time=399.6 us
4 KiB <<< . (ext4 /dev/dm-0 1.79 TiB): request=4 time=414.8 usThey are still approx 10 times higher than what other people report, but I'm not sure if that's due to the encryption.
dd results have not improved, though (well... they have, sort of, see below)
Edit:
It seems the improved ioping results are somewhat in line with other systems utilizing dm-crypt, for example https://lindevs.com/install-ioping-on-ubuntu/
Not sure about the actual hardware though, which makes it difficult to compare, but at least they are in the same ballpark.
dd results have not improved when using oflag=dsync, but when using conv=fsync as suggested by sabroad the results are now somewhat better:
# dd if=/dev/zero of=latency.img bs=4K count=1000 conv=fsync
4096000 bytes (4,1 MB, 3,9 MiB) copied, 0,0129761 s, 316 MB/sThere's much fluctuation though, but on average the MB/s count is a bit over 300.
Last edited by peabrain (2022-08-20 09:40:46)
Offline