[SOLVED] NVME unexpectedly low sequential read speed

Brocellous · 2023-07-16 05:19:48

I have a desktop here with two SSDs. On one is the ESP and linux install, and until recently the other held a mostly untouched Windows 11 installation from the manufacturer:

I was curious about the relative performance of these drives and tried the following test:

$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme0n1' # Unused windows drive
/dev/nvme0n1:
 Timing buffered disk reads: 9634 MB in  3.00 seconds = 3210.65 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 9620 MB in  3.00 seconds = 3206.00 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 9580 MB in  3.00 seconds = 3192.92 MB/sec

$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme1n1' # Linux drive
/dev/nvme1n1:
 Timing buffered disk reads: 4076 MB in  3.00 seconds = 1358.31 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4130 MB in  3.00 seconds = 1376.30 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4120 MB in  3.00 seconds = 1372.94 MB/sec

which is a pretty clear win for the first drive. In fact I was able to improve the speed a fair bit by reformatting it with the native block size and testing again:

$ sudo nvme format /dev/nvme0n1 -b 4096 # deletes windows and reformats /dev/nvme0n1
$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme0n1' # now unused drive
/dev/nvme0n1:
 Timing buffered disk reads: 12414 MB in  3.00 seconds = 4137.41 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 12686 MB in  3.00 seconds = 4228.17 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 12584 MB in  3.00 seconds = 4194.52 MB/sec

The first drive is now over 3x faster in this test. In researching more about the devices, it seems like the second drive is unexpectedly slow in this test. These are the models:

$ grep $ /sys/block/nvme*/device/{model,firmware_rev}
/sys/block/nvme0n1/device/model:PC SN810 NVMe WDC 2048GB # fast
/sys/block/nvme1n1/device/model:Sabrent Rocket 4.0 2TB # slow
/sys/block/nvme0n1/device/firmware_rev:61912523
/sys/block/nvme1n1/device/firmware_rev:RKT401.3

and their respective product pages from the manufacturer, I think:

/dev/nvme0n1: https://www.westerndigital.com/products … CPNRZ-2T00 (claims 6600MB/s seq read)
/dev/nvme1n1: https://sabrent.com/products/sb-rkt4p-2tb (claims 7000MB/s seq read)

I don't think either drive is artificially limited by the I/O, both are pcie gen 4 x4:

$ lspci -tvd::108 
-[0000:00]-+-06.0-[01]----00.0  Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD # /dev/nvme0n1, fast
           \-1a.0-[02]----00.0  Phison Electronics Corporation E16 PCIe4 NVMe Controller # /dev/nvme1n1, slow
$ grep $ /sys/block/nvme*/device/../../current_link_*(:P)
/sys/devices/pci0000:00/0000:00:06.0/0000:01:00.0/current_link_speed:16.0 GT/s PCIe
/sys/devices/pci0000:00/0000:00:06.0/0000:01:00.0/current_link_width:4
/sys/devices/pci0000:00/0000:00:1a.0/0000:02:00.0/current_link_speed:16.0 GT/s PCIe
/sys/devices/pci0000:00/0000:00:1a.0/0000:02:00.0/current_link_width:4

which iiuc implies a theoretical maximum transfer speed of 16e9*4/8*128/130 =~ 7500MB/s.

I don't expect these drives to reach their maximum performance in this test, but speed of the Sabrent drive (~1300 MB/s) is definitely lower than expected when the WD demonstrates ~4200MB/s is achievable in the same machine. I don't have a lot of experience benchmarking drives. Assuming the WD drive is working as intended, why might it be 3x as fast in this test? I thought the smaller block size might hurt the performance, but it didn't seem to change the WD drive speed by that much.

Last edited by Brocellous (2023-08-01 06:44:13)

seth · 2023-07-16 13:17:21

https://ssd-tester.de/sabrent_rocket_nvme_4_0_1tb.html has it at ~4GB/s

Sideload? Did you run the test from a system that booted from nvme1n1?
https://wiki.archlinux.org/title/Solid_ … ing_(APST) ?
=> https://wiki.archlinux.org/title/Solid_ … ST_support

I was able to improve the speed a fair bit by reformatting it with the native block size

I guess that's not an option for nvme1n1?

Brocellous · 2023-07-16 19:04:27

seth wrote:

Sideload? Did you run the test from a system that booted from nvme1n1?

Yes, I ran the test while booted from /dev/nvme1n1. I suppose this may affect the test.

seth wrote:

https://wiki.archlinux.org/title/Solid_ … ing_(APST) ?
=> https://wiki.archlinux.org/title/Solid_ … ST_support

Both devices have the same output for feature 0x0c, and no apparent errors.

seth wrote:

Brocellous wrote:
I was able to improve the speed a fair bit by reformatting it with the native block size
I guess that's not an option for nvme1n1?

It is, but obviously a bit more trouble considering it currently contains my install. Both drives have the same native block size:

$ sudo nvme id-ns -H /dev/nvme0n1 | tail -n2
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good 
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better (in use)
$ sudo nvme id-ns -H /dev/nvme1n1 | tail -n2 
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better

/dev/nvme1n1 is a 500MB ESP and the rest is btrfs on luks2. I haven't directly used btrfs send/receive before but it sounds like that would be a good way to copy my install from this drive to the other while I reformat it. If the other drive is just faster I might just leave it there, so I'd want to replicate the luks volume too. I will also want to change the luks sector size to match, though, as it is currently 512.

seth · 2023-07-16 19:30:36

I suppose this may affect the test.

https://grml.org/

Both devices have the same output for feature 0x0c, and no apparent errors.

Still try to simply disable it.

We'll postpone shredding your system a bit unless every other path has dried out

Brocellous · 2023-07-16 19:46:29

After booting with nvme_core.default_ps_max_latency_us=0

$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme0n1'
/dev/nvme0n1:
 Timing buffered disk reads: 12836 MB in  3.00 seconds = 4278.22 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 12882 MB in  3.00 seconds = 4293.74 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 12908 MB in  3.00 seconds = 4302.48 MB/sec
$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme1n1'
/dev/nvme1n1:
 Timing buffered disk reads: 4378 MB in  3.00 seconds = 1459.00 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4114 MB in  3.00 seconds = 1371.22 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4116 MB in  3.00 seconds = 1371.86 MB/sec

/dev/nvme1n1 had 1 ~90MB/s faster run, but otherwise the same. All /dev/nvme0n1 runs improved though, and it is ~100MB/s faster on average, a ~2% bump.

seth wrote:

We'll postpone shredding your system a bit unless every other path has dried out wink

Heh yeah. I would like to go through with reformatting it eventually because I expect the 512 -> 4096 formatting to help, but the current perf difference is a fair bit more than expected. Another thing I might try is physically swapping the nvme in the two m.2 slots, which shouldn't be too much trouble, though idk if that would have any effect.

seth · 2023-07-16 20:17:54

What evens out w/ the 6.5% bump in one pass.
The varying performance makes me wanna stress to try that offline to make sure there's no sideload.

Brocellous · 2023-07-25 04:07:35

Sorry for the delay. I was able to repeat the experiment from the archlinux iso:

# repeat 3 hdparm -t /dev/nvme0n1
/dev/nvme0n1:
 Timing buffered disk reads: 13348 MB in  3.00 seconds = 4448.84 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 13496 MB in  3.00 seconds = 4498.20 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 13352 MB in  3.00 seconds = 4450.58 MB/sec

# repeat 3 hdparm -t /dev/nvme1n1
/dev/nvme1n1:
 Timing buffered disk reads: 4152 MB in  3.00 seconds = 1383.89 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4220 MB in  3.00 seconds = 1406.34 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4124 MB in  3.00 seconds = 1374.15 MB/sec

It doesn't seem there is any change. I think the next step will have to be copying my system onto the other drive...

Brocellous · 2023-07-27 01:14:42

I've now copied my install from nvme1n1 to nvme0n1. I freshly partitioned /dev/nvme0n1 with the same layout then used rsync to copy the ESP, and cryptsetup + btrfs send/receive to recreate my luks volume and btrfs filesystem on top.

I now get the following interesting result from this test:

$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme0n1'
/dev/nvme0n1:
 Timing buffered disk reads: 6504 MB in  3.00 seconds = 2167.82 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 6616 MB in  3.00 seconds = 2205.14 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 6664 MB in  3.00 seconds = 2221.33 MB/sec
$ sudo -s <<< 'repeat 3 hdparm -t /dev/nvme1n1'
/dev/nvme1n1:
 Timing buffered disk reads: 4208 MB in  3.00 seconds = 1402.23 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 3920 MB in  3.00 seconds = 1306.45 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4166 MB in  3.00 seconds = 1388.63 MB/sec

The apparent seq read of nvme0n1 has roughly been cut in half for this test. It still shows faster than nvme1n1.

EDIT: and once more from the live environment of archlinux iso:

# repeat 3 hdparm -t /dev/nvme0n1
/dev/nvme0n1:
 Timing buffered disk reads: 6368 MB in  3.00 seconds = 2122.44 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 6582 MB in  3.00 seconds = 2193.75 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads: 6554 MB in  3.00 seconds = 2184.54 MB/sec
# repeat 3 hdparm -t /dev/nvme1n1
/dev/nvme1n1:
 Timing buffered disk reads: 4130 MB in  3.00 seconds = 1376.39 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 3958 MB in  3.00 seconds = 1318.83 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads: 4024 MB in  3.00 seconds = 1341.06 MB/sec

Last edited by Brocellous (2023-07-27 02:01:56)

seth · 2023-07-27 06:57:19

Does it simply depend on the disk usage (ratio)?
(nvme1n1 has basically not moved while nvme0n1 significantly dropped whey you - likely - put more data on it)

Brocellous · 2023-07-27 07:43:28

Yes, /dev/nvme0n1 was previously empty after having been formatted to 4k sector size. I'm not too familiar with the expected performance degradation, but these are both ~2T filesystems with about 500G used.

$ sudo btrfs filesystem usage -h /
Overall:
    Device size:		  1.86TiB
    Device allocated:		558.02GiB
    Device unallocated:		  1.32TiB
    Device missing:		    0.00B
    Device slack:		    0.00B
    Used:			552.15GiB
    Free (estimated):		  1.32TiB	(min: 679.35GiB)
    Free (statfs, df):		  1.32TiB
    Data ratio:			     1.00
    Metadata ratio:		     2.00
    Global reserve:		512.00MiB	(used: 0.00B)
    Multiple profiles:		       no

Data,single: Size:550.01GiB, Used:545.25GiB (99.14%)
   /dev/mapper/root	550.01GiB

Metadata,DUP: Size:4.00GiB, Used:3.45GiB (86.16%)
   /dev/mapper/root	  8.00GiB

System,DUP: Size:8.00MiB, Used:80.00KiB (0.98%)
   /dev/mapper/root	 16.00MiB

Unallocated:
   /dev/mapper/root	  1.32TiB

The new device should have been trimmed by both cryptsetup and mkfs.btrfs, so I don't expect a shortage of free blocks:

$ sudo cryptsetup status root 
/dev/mapper/root is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/nvme0n1p2
  sector size:  4096
  offset:  32768 sectors
  size:    3999737856 sectors
  mode:    read/write
  flags:   discards no_read_workqueue no_write_workqueue
$ grep $ /sys/block/nvme0n1/queue/discard_*         
/sys/block/nvme0n1/queue/discard_granularity:4096
/sys/block/nvme0n1/queue/discard_max_bytes:2199023255040
/sys/block/nvme0n1/queue/discard_max_hw_bytes:2
$ grep $ /sys/block/dm-0/queue/discard_* # /dev/nvme0n1p2
/sys/block/dm-0/queue/discard_granularity:4096
/sys/block/dm-0/queue/discard_max_bytes:2199023255040
/sys/block/dm-0/queue/discard_max_hw_bytes:2199023255040
/sys/block/dm-0/queue/discard_zeroes_data:0

I would also mostly expect it to affect write speeds, which I am not testing, but I did try "fstrim /" anyway with no apparent change in the speed test.

Brocellous · 2023-07-27 07:52:59

Sorry for double post, but from irc I tried and got the following interesting result:

$ sudo -s <<< 'repeat 3 hdparm -t --offset 1500 /dev/nvme0n1' 
/dev/nvme0n1:
 Timing buffered disk reads (offset 1500 GB): 12566 MB in  3.00 seconds = 4188.64 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads (offset 1500 GB): 12640 MB in  3.00 seconds = 4213.06 MB/sec
/dev/nvme0n1:
 Timing buffered disk reads (offset 1500 GB): 12466 MB in  3.00 seconds = 4154.75 MB/sec
$ sudo -s <<< 'repeat 3 hdparm -t --offset 1500 /dev/nvme1n1'
/dev/nvme1n1:
 Timing buffered disk reads (offset 1500 GB): 9708 MB in  3.00 seconds = 3235.70 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads (offset 1500 GB): 9882 MB in  3.00 seconds = 3293.72 MB/sec
/dev/nvme1n1:
 Timing buffered disk reads (offset 1500 GB): 9956 MB in  3.00 seconds = 3318.48 MB/sec

Since my luks volume has discards enabled (and idk what the on-disc format of btrfs is) I expect that these large offsets are (mostly?) empty. So it might be some kind of compression thing?

EDIT: I still plan to reformat the original drive 4k and test again, but I'll wait a little bit to be confident that I have copied everything correctly. Considering the above though, I think this is tentatively solved.

Last edited by Brocellous (2023-07-27 08:44:36)

Brocellous · 2023-08-01 06:43:11

I've now reformatted the original drive to the native 4k sector size, and switched to using a proper benchmarking tool, fio. I wasn't sure if it would matter in the seqread case, but in fact the queue depth is clearly necessary to saturate these drives.

Curiously, the device names (nvme0n1 & nvme1n1) swapped after formatting the original drive, so I'll refer to them by the model.

Using a fio job file like so:

[global]
name=seq-read
rw=read
time_based
ioengine=libaio
blocksize=1M
iodepth=64
direct=1
group_reporting
							
[seq-read-10]
runtime=10s
ramp_time=2s
numjobs=1

I am able to achieve the maximum speed (6600MB/s, even while mounted) exactly as advertised by the manufacturer with the WD SN810 device.

The Sabrent RKT4 device appears to max out around 5800MB/s even after reformatting it, which while short of the manufacturer's claims, is satisfactory for me. Marking solved.

Last edited by Brocellous (2023-08-01 06:43:54)

Arch Linux

#1 2023-07-16 05:19:48

[SOLVED] NVME unexpectedly low sequential read speed

#2 2023-07-16 13:17:21

Re: [SOLVED] NVME unexpectedly low sequential read speed

#3 2023-07-16 19:04:27

Re: [SOLVED] NVME unexpectedly low sequential read speed

#4 2023-07-16 19:30:36

Re: [SOLVED] NVME unexpectedly low sequential read speed

#5 2023-07-16 19:46:29

Re: [SOLVED] NVME unexpectedly low sequential read speed

#6 2023-07-16 20:17:54

Re: [SOLVED] NVME unexpectedly low sequential read speed

#7 2023-07-25 04:07:35

Re: [SOLVED] NVME unexpectedly low sequential read speed

#8 2023-07-27 01:14:42

Re: [SOLVED] NVME unexpectedly low sequential read speed

#9 2023-07-27 06:57:19

Re: [SOLVED] NVME unexpectedly low sequential read speed

#10 2023-07-27 07:43:28

Re: [SOLVED] NVME unexpectedly low sequential read speed

#11 2023-07-27 07:52:59

Re: [SOLVED] NVME unexpectedly low sequential read speed

#12 2023-08-01 06:43:11

Re: [SOLVED] NVME unexpectedly low sequential read speed

Board footer