Arch Linux

Cvlc

Member

Registered: 2020-03-26

Posts: 295

NVMe sector size / cryptsetup reencrypt

Hi,

My arch install is on a Btrfs subvolume inside a LUKS container, on an NVMe SSD. I cannot figure out if the sector sizes are correctly set :

# smartctl -a /dev/nvme0n1
Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

# fdisk -l /dev/nvme0n1
Disque /dev/nvme0n1 : 476,94 GiB, 512110190592 octets, 1000215216 secteurs
Modèle de disque : KINGSTON OM8PCP3512F-AI1                
Unités : secteur de 1 × 512 = 512 octets
Taille de secteur (logique / physique) : 512 octets / 512 octets
taille d'E/S (minimale / optimale) : 512 octets / 512 octets

lsblk -t
NAME          ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE  RA WSAME
zram0                 0   4096   4096    4096    4096    0           128 128    0B
nvme0n1               0    512      0     512     512    0 none     1023 128    0B
├─nvme0n1p1           0    512      0     512     512    0 none     1023 128    0B
└─nvme0n1p2           0    512      0     512     512    0 none     1023 128    0B
  └─cryptroot         0    512      0     512     512    0           128 128    0B

# cryptsetup luksDump /dev/nvme0n1p2
Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-xts-plain64
	sector: 512 [bytes]

I suppose that the correct sector size should be 4096 but it looks like it's 512 everywhere.
Should I just leave it as it is ? If I reinstall, what extra steps should I take in order to set everything to a sector size of 4096 ?

Bonus question: If I manage to set the ssd sector size correctly, since latest cryptsetup 2.4.0 automatically sets the sector size, would running "cryptsetup reencrypt" with no parameters automatically set the encrypted container sector size to 4096 as well ? What about Btrfs ?

Thanks very much !

Last edited by Cvlc (2021-08-28 13:47:16)

Cvlc

Member

Registered: 2020-03-26

Posts: 295

Re: NVMe sector size / cryptsetup reencrypt

Found this in the SSD page on the wiki

To change the sector size, use nvme format and specify the preferred value with the --lbaf parameter.

I suppose this should have been done prior to installing....

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: NVMe sector size / cryptsetup reencrypt

For cryptsetup, --sector-size is a new option and only supported with LUKS2. This affects how encryption is handled internally, the encryption result is different and so the on disk data is not compatible. Changing it will require re-encrypting everything.

Your device seems to be reporting 512 byte sector size logical/physical so that is what cryptsetup uses by default. I would not worry about it, and leave it as is, but I also don't have benchmarks.

Cvlc

Member

Registered: 2020-03-26

Posts: 295

Re: NVMe sector size / cryptsetup reencrypt

Thanks !

I'll format it properly if I reinstall. Just to confirm, would the steps be :

# nvme id-ns /dev/nvme0n1
...
lbaf  0 : ms:0   lbads:9  rp:0x2 (in use)
lbaf  1 : ms:0   lbads:12 rp:0x1 

# nvme format /dev/nvme0 --lbaf=1

check the correct new sector size is in use with #nvme id-ns
Then, if I understand correctly, all other levels should use proper sector size and alignment without any extra switches ? :

# fdisk /dev/nvme0n1
# cryptsetup luksFormat /dev/nvme0n1p2
# mkfs.btrfs -L Arch /dev/mapper/cryptroot

or should I use

# mkfs.btrfs -s 4096 /dev/mapper/mapped_device

as stated in dm-crypt page ?

Thanks for your help !

[edit]
about btrfs, I'm a little confused with the output of this command on my current setup :

# btrfs inspect-internal dump-super /dev/mapper/cryptroot | grep size
csum_size		4
sys_array_size		97
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
dev_item.sector_size	4096

How can btrfs be using 4096 sector size if the underlying structure isn't?
seems to be confirmed by

# blkid 
/dev/mapper/cryptroot: LABEL="Arch" UUID=".." UUID_SUB=".." BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme0n1p2: UUID=".." TYPE="crypto_LUKS" PARTLABEL="Linux LUKS" PARTUUID=".."
/dev/nvme0n1p1: SEC_TYPE="msdos" UUID=".." BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID=".."
...

Last edited by Cvlc (2021-08-28 09:16:03)

Cvlc

Member

Registered: 2020-03-26

Posts: 295

Re: NVMe sector size / cryptsetup reencrypt

(new post for clarity)

was curious and ran it in a vm, forcing virtual disk sector size with

<blockio logical_block_size='4096' physical_block_size='4096'/>

# fdisk /dev/vda
Disk /dev/vda : 20 GiB
Units : sectors of 1 × 4096 = 4096 bytes
Sector size (logical / physical) : 4096 bytes / 4096 bytes
I/O Size (Minimal/Optimal) :  4096 bytes / 4096 bytes

Device    Start     End  Sectors  Size Type
/dev/vda1   256   68815    66560  260M EFI System
/dev/vda2 68816 5242874  5176059 19,7G Linux filesystem

Cryptsetup and btrfs look like they're playing nicely.

But does the 256 start sector selected by fdisk look correct ? I thought it was 2048 by default

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: NVMe sector size / cryptsetup reencrypt

that's just regular MiB Alignment

2048 * 512 = 1048576
256 * 4096 = 1048576

note that GPT depends on sector size, so if you ever use this disk image with a logical sector size of 512, it will not work anymore (until you fix the partition table)

Last edited by frostschutz (2021-08-28 11:03:43)

Cvlc

Member

Registered: 2020-03-26

Posts: 295

Re: NVMe sector size / cryptsetup reencrypt

now I get it

Thanks very much, solved then. Only thing I have to do is

# nvme format /dev/nvme0 --lbaf=1

before next install.

Have a good day !

Last edited by Cvlc (2021-08-28 13:54:31)

Cvlc

Member

Registered: 2020-03-26

Posts: 295

Re: NVMe sector size / cryptsetup reencrypt

For cryptsetup, --sector-size is a new option and only supported with LUKS2. This affects how encryption is handled internally, the encryption result is different and so the on disk data is not compatible. Changing it will require re-encrypting everything.

Just tested that out in a VM :
by default, cryptsetup reencrypt (v2.4.0) retains the previous sector size, and will not automatically change it to 4096 (whereas cryptsetup luksFormat will autodetect the correct sector size). However cryptsetup reencrypt does accept the --sector-size=4096 parameter, which allows to reencrypt without data loss.

[EDIT]
So I guess it's not solved yet then !
Apparently my btrfs filesystem does use 4096 byte sectors (see above). So I could theoretically run cryptsetup reencrypt --sector-size=4096 ... but should I ?
It works in the VM without data loss, but I am reluctant to try it on my laptop...
How bad performance-wise is it to be running a filesystem using 4K sectors on top of a luks container using 512b sectors, on top of a transition layer from 512 to 4K physical sectors....? it looks like a mess !

Last edited by Cvlc (2021-08-28 14:14:33)