[SOLVED] Massive performance boost when using the nobarrier mount opt.

Pavle · 2022-06-04 16:54:12

Summary

Mounting an SSD with `nobarrier` option results in massive improvement in performance for MariaDB on one machine, but makes no difference at all on another, similar machine. At the same time, results from `hdparm -Tt` are hardly affected by this mount option.

The whole story

Recently I switched from Thinkpad T520 to Thinkpad W520 (the latter being twice as powerful in every respect) and immediately noticed a significant reduction in performance when using MariaDB. Both laptops have SSD storage. After some investigation and experimentation, it turned out that setting the `nobarrier` mount option fixed this problem, which is expected and documented. However, setting this option makes no difference on my old laptop (T520). How can this be? Are there some other config options that might be affecting SSD performance? Furthermore, if this option makes such a drastic improvement, I think it should be documented in the arch install manual, or at least the MariaDB page.

Metrics

Here are some metrics I collected on both machines (all averaged over 3 executions):

MariaDB import SQL dump:

T520 without nobarrier: 24.19 seconds
T520 with nobarrier: 23.85 seconds

W520 without nobarrier: 99.67 seconds
W520 with nobarrier: 22.91 seconds

`hdparm -Tt`:

T520 without nobarrier: a) cached reads: 4785.46 MB/s b) buffered reads: 257.2 MB/s
T520 with nobarrier: a) cached reads: 4923.89 MB/s b) buffered reads: 256.78 MB/s

W520 without nobarrier: a) cached reads: 8804.76 MB/s b) buffered reads: 530.76 MB/s
W520 with nobarrier: a) cached reads: 8337.92 MB/s b) buffered reads: 528.64 MB/s

It's worth noting that the performance actually dropped (at least according to hdparm) when setting the nobarrier option, while the real world performance skyrocketed. I'd also point out that I haven't noticed any sluggishness while copying files on the filesystem, but I didn't run any benchmarks to confirm (I could add that in a comment later).

SSD info:

T520: KINGSTON SMS200S3120G
W520: SAMSUNG MZ7TD256HAFV-000L9

Comparing the output of `hdparm -I` on the two machines, some notewothy differences are:

- `Advanced power management level` on T520 set to 254, unset on W520
- `Look-ahead` enabled on W520, disabled on T520
- `Device encrypts all user data` enabled on W520, not even listed on T520 (this one seems a likely culprit)

--------------

What I'm trying to do here is first to understand what is happening here, what's causing the striking differences between two machines and to gain better understanding of tuning SSD performance on arch. So I'd like to understand why the T520 laptop performed better out-of-the box than the much more powerful W520. I'd like to understand what other config options might be affecting this performance difference. Finally, I'd like to share my experience with others who might be having a similar issue, so that at least they have a possible fix. Any input from the community is welcome. Let me know if you need other specs.

Thank you!

Last edited by Pavle (2022-06-07 17:26:14)

dimich · 2022-06-04 21:16:42

I can't say anything about MariaDB performance, but it's not surprising that nobarrier mount option does not affect hdparm raw block device test.

Pavle · 2022-06-05 06:53:01

Thanks! Would you mind explaining why that is? Also, which tool would be more suitable for measuring SSD performance?

seth · 2022-06-05 07:38:04

Because it operates below the FS (and read only)

barriers protect data integrity against immediate power losses on volatile drive caches, https://unix.stackexchange.com/question … n-in-linux
Deactivating them gives you more performance because the drive is actually writing less - on the risk that a power loss may severely corrupt the FS.

The significant difference may be caused by
a) really shitty writing performance of the new drive (possibly encryption related)
b) more flushing requirements (effectively sideload) on the new system

Measuring the drive speed itself
a) has to happen offline (the drive must not be used by the OS or even mounted)
b) raw (not involving the FS, hence not mounted)

You could use dd for that, but that's gonna wipe the drive.
badblocks has a non-destructive test mode that reads and writes back blocks that you could time.

Pavle · 2022-06-05 15:28:22

Any ideas how I can disable this encryption feature? BTW, I don't think encryption is actually enabled, because if that were the case, I would presumably need to enter a password during BIOS boot stage. So I'm a bit confused about this "Device encrypts all user data" option being enabled. Perhaps that simply indicates that the SSD is capable of self-encryption? In BIOS there is an option to configure a password for the hard disk, but that's currently disabled.

dimich · 2022-06-05 16:15:09

I doubt hardware enryption on device itself significantly affects filesystem performance but doesn't affect raw reads. What filesystems are on both machines? What other mount options? Write caching and readahead options?

Pavle · 2022-06-05 17:14:01

Filesystem is ext4 on both machines.

Mount options:

T520: rw,relatime,data=ordered
W520: rw,relatime,nobarrier

(I'll test the data=ordered option on W520 and report back)

As for other options, here is the output of `hdparm -I` for both drives:

T520:

/dev/sdb:

ATA device, with non-removable media
	Model Number:       KINGSTON SMS200S3120G                   
	Serial Number:      50026B7272038E5D    
	Firmware Revision:  60AABBF0
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x0110) 
	Supported: 8 7 6 5 
	Likely used: 8
Configuration:
	Logical		max	current
	cylinders	16383	16383
	heads		16	16
	sectors/track	63	63
	--
	CHS current addressable sectors:    16514064
	LBA    user addressable sectors:   234441648
	LBA48  user addressable sectors:   234441648
	Logical  Sector size:                   512 bytes
	Physical Sector size:                   512 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:      114473 MBytes
	device size with M = 1000*1000:      120034 MBytes (120 GB)
	cache/buffer size  = unknown
	Nominal Media Rotation Rate: Solid State Device
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, with device specific minimum
	R/W multiple sector transfer: Max = 1	Current = 1
	Advanced power management level: 254
	DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	    	Look-ahead
	   *	Host Protected Area feature set
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	   *	Advanced Power Management feature set
	    	Power-Up In Standby feature set
	   *	SET_FEATURES required to spinup after power up
	    	SET_MAX security extension
	   *	48-bit Address feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	WRITE_{DMA|MULTIPLE}_FUA_EXT
	   *	64-bit World wide name
	   *	IDLE_IMMEDIATE with UNLOAD
	    	Write-Read-Verify feature set
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	   *	unknown 119[6]
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Gen3 signaling speed (6.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Phy event counters
	   *	NCQ priority information
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	   *	DMA Setup Auto-Activate optimization
	   *	Device-initiated interface power management
	   *	Software settings preservation
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Write Same (AC2)
	   *	SCT Data Tables (AC5)
	   *	Extended number of user addressable sectors 
	   *	DOWNLOAD MICROCODE DMA command
	   *	SET MAX SETPASSWORD/UNLOCK DMA commands
	   *	WRITE BUFFER DMA command
	   *	READ BUFFER DMA command
	   *	Data Set Management TRIM supported (limit 1 block)
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
		frozen
	not	expired: security count
		supported: enhanced erase
	2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50026b7272038e5d
	NAA		: 5
	IEEE OUI	: 0026b7
	Unique ID	: 272038e5d
Checksum: correct

W520:

/dev/sda:

ATA device, with non-removable media
	Model Number:       SAMSUNG MZ7TD256HAFV-000L9              
	Serial Number:      S17LNSADB02584      
	Firmware Revision:  DXT04L6Q
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x0039) 
	Supported: 9 8 7 6 5 
	Likely used: 9
Configuration:
	Logical		max	current
	cylinders	16383	65520
	heads		16	1
	sectors/track	63	63
	--
	CHS current addressable sectors:     4127760
	LBA    user addressable sectors:   268435455
	LBA48  user addressable sectors:   500118192
	Logical  Sector size:                   512 bytes
	Physical Sector size:                   512 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:      244198 MBytes
	device size with M = 1000*1000:      256060 MBytes (256 GB)
	cache/buffer size  = unknown
	Nominal Media Rotation Rate: Solid State Device
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, no device specific minimum
	R/W multiple sector transfer: Max = 1	Current = 1
	DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	Host Protected Area feature set
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	DOWNLOAD_MICROCODE
	    	SET_MAX security extension
	   *	48-bit Address feature set
	   *	Device Configuration Overlay feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	WRITE_{DMA|MULTIPLE}_FUA_EXT
	   *	64-bit World wide name
	    	Write-Read-Verify feature set
	   *	WRITE_UNCORRECTABLE_EXT command
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Gen3 signaling speed (6.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Host-initiated interface power management
	   *	Phy event counters
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	   *	DMA Setup Auto-Activate optimization
	    	Device-initiated interface power management
	    	Asynchronous notification (eg. media change)
	   *	Software settings preservation
	    	Device Sleep (DEVSLP)
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Write Same (AC2)
	   *	SCT Error Recovery Control (AC3)
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	   *	Device encrypts all user data
	   *	IEEE 1667 authentication of transient storage devices
	   *	DOWNLOAD MICROCODE DMA command
	   *	SET MAX SETPASSWORD/UNLOCK DMA commands
	   *	WRITE BUFFER DMA command
	   *	READ BUFFER DMA command
	   *	Data Set Management TRIM supported (limit 8 blocks)
	   *	Deterministic read ZEROs after TRIM
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
		frozen
	not	expired: security count
		supported: enhanced erase
	2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5002538500000000
	NAA		: 5
	IEEE OUI	: 002538
	Unique ID	: 500000000
Device Sleep:
	DEVSLP Exit Timeout (DETO): 20 ms (drive)
	Minimum DEVSLP Assertion Time (MDAT): 30 ms (drive)
Checksum: correct

Hopefully this contains the caching / readahead options you asked for. If not, please let me know how to obtain those. Thanks!

Last edited by Pavle (2022-06-05 17:14:53)

Pavle · 2022-06-05 17:31:22

Tested W520 with mount option `data=ordered` and it didn't make any difference. Performance is identical as without the flag. Curiously, `mount -l` doesn't show the new mount option. On the T520 machine, it does show it. Strange...

seth · 2022-06-06 07:30:09

seth wrote:

Measuring the drive speed itself
a) has to happen offline (the drive must not be used by the OS or even mounted)
b) raw (not involving the FS, hence not mounted)
You could use dd for that, but that's gonna wipe the drive.
badblocks has a non-destructive test mode that reads and writes back blocks that you could time.

Your original tests were a hdparm read and what I understand to be an sql dump to file (ie. write) - only the writing action was impacted by "nobarrier" what's kinda expectable but that also means that we've metrcis for
* filesystem write
* raw read
what's not significant to point to either FS or drive.

If you only care about reading speeds
a) that's not supposed to be affected by nobarrier (unless the device is busy writing during the test)
b) you can dd a file/bytes *from* the drive into /dev/null

Curiously, `mount -l` doesn't show the new mount option. On the T520 machine, it does show it.

ordered is the default anyway - how exactly do you add the option and mount the device? Is this still w/ "nobarrier"?

Pavle · 2022-06-06 08:16:29

ordered is the default anyway - how exactly do you add the option and mount the device? Is this still w/ "nobarrier"?

No, I removed the `nobarrier` option, so the entry in fstab looked like this:

UUID=...       /               ext4            rw,relatime,data=ordered   0 1

seth · 2022-06-06 13:58:54

What's the complete fstab (the UUIDs are not secret, they exist to prevent collisions) and "mount" ouptut?

Pavle · 2022-06-06 15:33:36

Contents of /etc/fstab with nobarrier option set:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda2
UUID=ae9b6187-97e9-40c2-ab0d-339e060a95b4	/         	ext4      	rw,relatime,nobarrier	0 1

# /dev/sda1
UUID=198F-0FA5      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

Output of `mount -l | grep sda` with this fstab applied (after reboot):

/dev/sda2 on / type ext4 (rw,relatime,nobarrier)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

Contents of /etc/fstab with `data=ordered` option set:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda2
UUID=ae9b6187-97e9-40c2-ab0d-339e060a95b4	/         	ext4      	rw,relatime,data=ordered	0 1

# /dev/sda1
UUID=198F-0FA5      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

Output of `mount -l | grep sda` with this fstab applied (after reboot):

/dev/sda2 on / type ext4 (rw,relatime)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

seth · 2022-06-06 15:46:07

Interestingly doesn't show up here either

systemctl -- show '-.mount'

suggests that systemd fixes filters that "for you", but as mentioned, it's the default setting anyway.

Did you try offline read/write performance tests?

Pavle · 2022-06-07 06:49:29

So I did the non-destructive badblocks test. I booted into the arch ISO image on USB drive and performed the test from there. The test on the smaller partition (/boot) finished in no time and on the larger partition (/) it took 4 hours, 10 minutes and 52 seconds. The size of that partition is 234G. Not sure what this tells us, unless your idea was to perform the same test on both machines and compare results. However, the disk in the other machine is significantly smaller, so it wouldn't really be a fair comparison.

seth · 2022-06-07 06:55:19

badblocks allows you to limit the range of tested blocks and yes: the idea was to compare both machines to get an idea about their comparative raw disk performance (below the FS)
Btw:

smaller partition (/boot) finished in no time and on the larger partition (/)

sounds a bit like those were mounted during the test. In that case and again: you need to run this offline (eg. from a live grml or so)

Pavle · 2022-06-07 07:23:16

sounds a bit like those were mounted during the test. In that case and again: you need to run this offline (eg. from a live grml or so)

No, no, they were definitely not mounted. As I said, I booted into the arch ISO image on a USB drive. The boot partition is only 300M, so it makes sense that it was done very quickly.

Pavle · 2022-06-07 13:33:52

I ran badblocks on T520 machine and here are the results:

- 20GB partition took 9 minutes 39 seconds
- 82GB partition took 40 minutes 16 seconds

As I said earlier, I'm not sure what this tells us, given the size difference. We could perhaps extrapolate that the speed is roughly 29 seconds per GB. By the same token, the speed of the SSD in W520 machine is 64 seconds per GB. This would suggest that the latter SSD is simply much slower. But I'm not sure this assumption is valid.

seth · 2022-06-07 13:53:27

seth wrote:

badblocks allows you to limit the range of tested blocks

But yes, this suggests that the new drive is slower (or more cache reliant) and a random google result
https://www.harddrivebenchmark.net/hdd. … 0G&id=5213
https://www.harddrivebenchmark.net/hdd. … L9&id=5946
seems to mildly support that, too (we don't know the exact metric here)

Pavle · 2022-06-07 14:18:11

badblocks allows you to limit the range of tested blocks

Could you please help me with this? I can see that there is a `-b` option, which controls the block size. But I don't see anything that controls the range of tested blocks.

seth · 2022-06-07 14:55:45

https://man.archlinux.org/man/core/e2fsprogs/badblocks.8.en wrote:

badblocks [ -svwnfBX ] [ -b block_size ] [ -c blocks_at_once ] [ -d read_delay_factor ] [ -e max_bad_blocks ] [ -i input_file ] [ -o output_file ] [ -p num_passes ] [ -t test_pattern ] device [ last_block ] [ first_block ]

Pavle · 2022-06-07 17:24:24

Not knowing which values you're supposed to provide for `last_block` and `first_block`, I experimented a bit and ended up running the command like so:

badblocks -nsv /dev/sda2 1000000 0

which, in all likelihood, tested the first gigabyte on the partition (but this is guesswork). In any case, this took only ~28 seconds on T520 and 1 minute 36 seconds on W520! So it seems that there is a big difference in performance between the two disks. Here's a nice performance comparison: https://ssd.userbenchmark.com/Compare/K … 536vsm2342.

So I guess we can close this one as hardware-related. A big thank you to seth for helping me with this issue!

seth · 2022-06-07 20:16:03

Ftr, a block is 512 bytes (on this drive - see #7), so you tested ~488 MB (but that doesn't matter - it was the same amount on both drives and you keep getting 3-4 times less throughput on raw writes.

Arch Linux

#1 2022-06-04 16:54:12