nvme + dm-crypt - question regarding vector size

Utini · 2023-08-23 17:29:11

I am using LUKS 2.x on a Solidigm P44 Pro 2TB NVME.
I did the standard dm-crypt setup and didn't do anything to vector sizes so far.

LUKS 2.x should automatically choose the best vector size but I am unsure if this is really the case for me?

fdisk -l

Disk /dev/nvme0n1: 1,86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk model: SOLIDIGM x               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: x

Device              Start        End    Sectors   Size Type
/dev/nvme0n1p1       2048     206847     204800   100M EFI System
/dev/nvme0n1p2     206848     239615      32768    16M Microsoft reserved
/dev/nvme0n1p3     239616 1047532172 1047292557 499,4G Microsoft basic data
/dev/nvme0n1p4 1047533568 1048575999    1042432   509M Windows recovery environment
/dev/nvme0n1p5 1048576000 1049599999    1024000   500M Linux extended boot
/dev/nvme0n1p6 1049600000 4000796671 2951196672   1,4T Linux filesystem


Disk /dev/mapper/lvm: 1,37 TiB, 1510995918848 bytes, 2951163904 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/MyVolume: 1,37 TiB, 1510456950784 bytes, 2950111232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 15,06 GiB, 16173236224 bytes, 3948544 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

cryptsetup luksDump /dev/nvme0n1p6

LUKS header information
Version:        2
Epoch:          11
Metadata area:  
Keyslots area:  
UUID:          
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt

        length: (whole device)
        cipher: aes-xts-plain64
        sector: 512 [bytes]

Keyslots:
  0: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      argon2id
        Time cost:  9
        Memory:     1048576
        Threads:    4
        AF hash:    sha256

Tokens:
Digests:
  0: pbkdf2
        Hash:       sha256
        Iterations: 329740

nvme id-ns -H /dev/nvme0n1 | grep "Relative Performance"

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)

smartctl -a /dev/nvme0n1

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.4.11-arch2-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

Namespace 1 Formatted LBA Size:     512
Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

cryptsetup luksDump /dev/nvme0n1p6 | grep sector

sector: 512 [bytes]

So far it looks like 512 bytes is the best/correct setting and 512 is being used.
How ever:

tune2fs -l /dev/mapper/MyVolume

Block size:               4096
Fragment size:            4096

Does that mean I am using 4096 bytes after all?
I was also under the impression that 4096 bytes increased performance, so should I make a change to it if it is not used already?

Thanks!

Btw, by posting this I realized that I disabled discard at some point.
I now set the correct flags for discard + no-read-workqueue no-write-workqueue

frostschutz · 2023-08-23 18:38:11

Your LUKS header uses 512 byte sector size. It could be using 4096 byte sectors instead (provided the device is a multiple of 4096 bytes large, or else it stops working altogether).

LUKS sector size does not affect alignment (that depends on partition offset, data offset and the filesystem itself), and does not affect performance too much either (last time I tested anyway).

Changing the LUKS sector size would require you to re-encrypt all data (cryptsetup reencrypt --sector-size). If you don't have any specific problems with the current setup, just leave as is.

Utini · 2023-08-24 05:05:59

Hi again,

well I am trying to get the most performance out of my device while using LUKS since I have a speedy laptop + NVME.
So far I have been trying to use "no_read_workqueue and no_write_workqueue" flags while using benchmark tools like kdediskmark and simple stuff like:

fio --filename=blyat --readwrite=read --bs=1m --direct=1 --loops=10000 -runtime=1m --name=plain --size=1g

But my results are always a bit weird and mixed.

For example without "no_read_workqueue and no_write_workqueue":

Read:

KDiskMark SEQ 1MiB/8Queues/1Thread: 5.911,68 MB/s
KDiskMark SEQ 1MiB/1Queues/1Thread: 996,03 MB/s
KDiskMark RND 4KiB/32Queues/1Thread: 243,19 MB/s
KDiskMark RND 4KiB/1Queues/1Thread: 43,01 MB/s

Write:

KDiskMark SEQ 1MiB/8Queues/1Thread: 2.164,93 MB/s
KDiskMark SEQ 1MiB/1Queues/1Thread: 679,14 MB/s
KDiskMark RND 4KiB/32Queues/1Thread: 405,83 MB/s
KDiskMark RND 4KiB/1Queues/1Thread: 84,66 MB/s

And another write benchmark with only the first two write tests:

KDiskMark SEQ 1MiB/8Queues/1Thread: 3.716,93 MB/s
KDiskMark SEQ 1MiB/1Queues/1Thread: 706,92MB/s

With "no_read_workqueue and no_write_workqueue" set:

Read:

KDiskMark SEQ 1MiB/8Queues/1Thread: 1.223,86 MB/s
KDiskMark SEQ 1MiB/1Queues/1Thread: 893,86 MB/s
KDiskMark RND 4KiB/32Queues/1Thread: 162,37 MB/s
KDiskMark RND 4KiB/1Queues/1Thread: 50,86 MB/s

Write:

KDiskMark SEQ 1MiB/8Queues/1Thread: 999,21 MB/s
KDiskMark SEQ 1MiB/1Queues/1Thread: 926,59MB/s
KDiskMark RND 4KiB/32Queues/1Thread: 164,83 MB/s
KDiskMark RND 4KiB/1Queues/1Thread: 133,10 MB/s

So the results are all over the place and I am not sure how to even interpret them.
The biggest different is the SEQ 1MiB/8Queues/1Thread read + write which seems to be huge!

How ever running this benchmark doesn't show too much differences in a simple read/write test:

fio --filename=bla --readwrite=write --bs=1m --direct=1 --loops=10000 -runtime=3m --name=plain --size=1g  
fio --filename=bla --readwrite=read--bs=1m --direct=1 --loops=10000 -runtime=3m --name=plain --size=1g

Without "no_read_workqueue and no_write_workqueue":

WRITE: bw=830MiB/s (870MB/s), 830MiB/s-830MiB/s (870MB/s-870MB/s), io=48.6GiB (52.2GB)
READ: bw=1055MiB/s (1107MB/s), 1055MiB/s-1055MiB/s (1107MB/s-1107MB/s), io=61.8GiB (66.4GB)

With "no_read_workqueue and no_write_workqueue" set:

 WRITE: bw=988MiB/s (1036MB/s), 988MiB/s-988MiB/s (1036MB/s-1036MB/s), io=174GiB (186GB)
  READ: bw=914MiB/s (959MB/s), 914MiB/s-914MiB/s (959MB/s-959MB/s), io=53.6GiB (57.5GB)

So this is the first part that confuses me already.
By researching it I found several reports/benchmarks (on reddit and other distro communities) where increasing the sector size from 512 to 4096 bytes helped a lot as well.
The only downside to this (from my research) was, that a power loss could cause a bigger data loss.
But I am on a laptop and would automatically correctly shut down / suspend at a critical level anyway.

So I was hoping to be able to clarify and figuring out the best settings here

Last edited by Utini (2023-08-24 05:07:18)

Arch Linux

#1 2023-08-23 17:29:11

nvme + dm-crypt - question regarding vector size

#2 2023-08-23 18:38:11

Re: nvme + dm-crypt - question regarding vector size

#3 2023-08-24 05:05:59

Re: nvme + dm-crypt - question regarding vector size

Board footer