[SOLVED] SSD write speeds seems to be limited to 30 MB/s

Amplificator · 2017-09-09 17:02:39

Hi.

I have 2 ssd's in my system, one is a 64GB Crucial M4 that I use for my root and home partitions and the other is a 1TB Crucial MX300 which I use for other things such as VM's.
I then have 3 hdd's as well that I use for.. things.

The problem is that any ssd I attach to my system (I have tried several more) is somehow limited to about 30 mb/s write speeds.

Here's a dd write test on my MX300 disk:

martin@xws:/volumes/mx300/TEST|⇒  dd if=/dev/zero of=test bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 34,8714 s, 30,8 MB/s

I have tried to test with fio as well and it confirms the speed.

The hdd's perform fine and I can easily achieve above 100mb/s write on those.

I can't remember the ssd write speeds ever being not-limited to the 30mb/s but for a long time I just simply ignored it because it was fine for my system disk, as I would build packages in /tmp anyway.
But then I got the 1TB ssd and now I'd obviously like all the speed I can get out of it.

The "funny" thing is that if I boot any other OS on my system, such as a Fedora live cd and then mount both of my ssd's I can get perfectly fine speeds (though I'm limited to SATA2 but that is fine by me), which means it's an issue with my Arch install.

If I run the same command on my MX300 in the Fedora 26 live cd I get:

[root@localhost-live test]# dd if=/dev/zero of=test bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.40942 s, 244 MB/s

..and the same holds true with my M4 drive; its performance is just fine outside of Arch.

I do not use LVM at all, so that can't be an issue.

What I have tried:
1. Updating firmware for everything - bios, drives and sata controllers. Software packages is also up to date in Arch.
2. I have tried two different controllers, one being the integrated Intel controller and the other being an LSI SAS controller which is also integreted in my motherboard.
3. I have tried changing cables.
4. I have checked bios settings for options that could have an impact.
5. Tried various boot options in fstab after reading XFS documentation (I use XFS for all my drives). I also tried the exact same boot option that Fedora live cd mounts it with.
6. Tried several IO schedulers (noop, deadline, cfq and bfq)
7. Tried several kernels. I have always been running linux-zen but I have tried the stock kernel as well as the linux-lts kernel.
8. I have changed write caching using hdparm.
9. Tried various sysctl settings, ranging from completely stock/clean, to my own settings and to the exact same settings that the Fedora live iso is using.

My /etc/fstab:

/dev/disk/by-uuid/98d99355-a8d8-4aea-9446-63a841f987da /boot ext2 rw,relatime,barrier,user_xattr,acl 0 2
/dev/disk/by-uuid/77ed90e6-6f1c-417f-9647-6bc0e7804482 /volumes/m4 xfs rw,relatime,attr2,barrier,nodiscard,inode64,noquota,x-gvfs-show,x-gvfs-name=M4,x-gvfs-icon=drive-harddisk,x-gvfs-symbolic-icon=drive-harddisk 0 1
/dev/disk/by-uuid/c3c731a7-9c48-49a8-9a39-ac0f372bb7a2 /volumes/mx300 xfs rw,relatime,attr2,barrier,nodiscard,inode64,noquota,x-gvfs-show,x-gvfs-name=MX300,x-gvfs-icon=drive-harddisk,x-gvfs-symbolic-icon=drive-harddisk 0 0
/dev/disk/by-uuid/e006f95e-1940-4aef-a9b5-9289f6e91c69 /volumes/velociraptor xfs rw,relatime,attr2,barrier,nodiscard,inode64,noquota,x-gvfs-show,x-gvfs-name=VelociRaptor,x-gvfs-icon=drive-harddisk,x-gvfs-symbolic-icon=drive-harddisk 0 0
/dev/disk/by-uuid/4dd972d4-c6a7-4ff8-9b08-5905ee71810d /volumes/storage xfs rw,relatime,attr2,barrier,nodiscard,inode64,noquota,x-gvfs-show,x-gvfs-name=Storage,x-gvfs-icon=drive-harddisk,x-gvfs-symbolic-icon=drive-harddisk 0 0
/dev/disk/by-uuid/302c88cf-e3ed-4de5-a589-6b3fad4d2867 /volumes/cellar xfs rw,relatime,attr2,barrier,nodiscard,inode64,noquota,x-gvfs-show,x-gvfs-name=Cellar,x-gvfs-icon=drive-harddisk,x-gvfs-symbolic-icon=drive-harddisk 0 0

My sysctl settings:

###
### GENERAL SYSTEM SECURITY OPTIONS ###
###

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Allow for more PIDs
kernel.pid_max = 262144

# Enable randomization of Address Space Layout Randomization to protect against certain buffer overflow attacks.
kernel.randomize_va_space = 2

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65535

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65535

# Restrict core dumps
fs.suid_dumpable = 0

# Hide exposed kernel pointers
kernel.kptr_restrict = 1

# Restrict access to kernel logs
kernel.dmesg_restrict = 1


###
### IMPROVE SYSTEM MEMORY MANAGEMENT ###
###

# Do less swapping
vm.swappiness = 30
vm.dirty_ratio = 5
vm.dirty_background_ratio = 3
vm.vfs_cache_pressure = 50

# Specifies the minimum virtual address that a process is allowed to mmap
vm.mmap_min_addr = 4096

# 50% overcommitment of available memory
vm.overcommit_memory = 0
vm.overcommit_ratio = 50

# Set maximum amount of memory allocated to shm to 256MB
kernel.shmmax = 268435456
kernel.shmall = 268435456

# Keep at least 256MB of free memory space available
vm.min_free_kbytes = 262144

# Reserve 33GB of memory to HugePages for more efficient VM memory management
vm.nr_hugepages = 16896

# Fix small periodic system freezes
vm.dirty_background_bytes = 8388608
vm.dirty_bytes = 8388608

# Increase size of file handles, inode cache and inotify cache
fs.file-max = 524288
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 2048
fs.inotify.max_queued_events = 32768


###
### GENERAL NETWORK SECURITY OPTIONS ###
###

# Prevent SYN attack, enable SYNcookies (they will kick-in when the max_syn_backlog reached)
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 8192

# Enable packet forwarding
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1

# Disable IP source routing
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 10

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

# Don't relay bootp
net.ipv4.conf.all.bootp_relay = 0

# Don't proxy arp for anyone
net.ipv4.conf.all.proxy_arp = 0

# Turn on the tcp_timestamps, accurate timestamp make TCP congestion control algorithms work better
net.ipv4.tcp_timestamps = 1

# Don't ignore directed pings
net.ipv4.icmp_echo_ignore_all = 0

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

## Since we are not a router we disable redirects
net.ipv4.conf.all.send_redirects = 0

# Allowed local port range
net.ipv4.ip_local_port_range = 1024 65535

# Enable a fix for RFC1337 - time-wait assassination hazards in TCP
net.ipv4.tcp_rfc1337 = 1

# Do not auto-configure IPv6
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_ra=0


###
### TUNING NETWORK PERFORMANCE ###
###

# For high-bandwidth low-latency networks, use 'htcp' congestion control
# For general usage use, use 'westwood' or 'reno' congestion control
# If available, use 'yeah' congestion control
# Depending on your choice, do a 'modprobe tcp_htcp', 'modprobe tcp_westwood', 'modprobe tcp_reno' or 'modprobe tcp_yeah' first
net.ipv4.tcp_congestion_control = westwood

# For servers with tcp-heavy workloads, enable 'fq' queue management scheduler (kernel > 3.12)
net.core.default_qdisc = fq

# Turn on the tcp_window_scaling
net.ipv4.tcp_window_scaling = 1

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
net.core.rmem_default = 262144
net.core.rmem_max = 16777216

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384
net.core.wmem_default = 262144
net.core.wmem_max = 16777216

# Increase number of incoming connections
net.core.somaxconn = 32768

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 16384
net.core.dev_weight = 128

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 65535

# Increase the maximum total buffer-space allocatable. This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000

# Try to reuse time-wait connections, but don't recycle them (recycle can break clients behind NAT)
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0

# Limit number of orphans, each orphan can eat up to 16M (max wmem) of unswappable memory
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_orphan_retries = 0

# Increase the maximum memory used to reassemble IP fragments
#net.ipv4.ipfrag_high_thresh = 524288
#net.ipv4.ipfrag_low_thresh = 262144

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

# Increase size of RPC datagram queue length
net.unix.max_dgram_qlen = 50

# Don't allow the arp table to become bigger than this
net.ipv4.neigh.default.gc_thresh3 = 2048

# Tell the gc when to become aggressive with arp table cleaning.
# Adjust this based on size of the LAN. 1024 is suitable for most /24 networks
net.ipv4.neigh.default.gc_thresh2 = 1024

# Adjust where the gc will leave arp table alone - set to 32.
net.ipv4.neigh.default.gc_thresh1 = 32

# Adjust to arp table gc to clean-up more often
net.ipv4.neigh.default.gc_interval = 30

# Increase TCP queue length
net.ipv4.neigh.default.proxy_qlen = 96
net.ipv4.neigh.default.unres_qlen = 6

# Enable Explicit Congestion Notification (RFC 3168), disable it if it doesn't work for you
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_reordering = 3

# How many times to retry killing an alive TCP connection
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15

# Avoid falling back to slow start after a connection goes idle
# keeps our cwnd large with the keep alive connections (kernel > 3.6)
net.ipv4.tcp_slow_start_after_idle = 0

# Allow the TCP fastopen flag to be used, beware some firewalls do not like TFO! (kernel > 3.7)
net.ipv4.tcp_fastopen = 3

# This will enusre that immediatly subsequent connections use the new values
net.ipv4.route.flush = 1
net.ipv6.route.flush = 1

Mount options and sysctl settings from the Fedora 26 live cd can be found here: https://ptpb.pw/YyJb/bash but as I said I've tried those exact settings as well and no difference.

Since the write speeds are fine in everyOS other than my Arch installation (I have tried several linux distros and even Windows as well) I'm faily convinced that the problem is some obscure random silly setting that someone put there just to make me unhappy

So at this point I'm all out of ideas. Ask for more info if needed.

Last edited by Amplificator (2017-09-10 16:01:21)

Amplificator · 2017-09-10 16:04:09

Alright, I fixed it myself and I feel as stupid as ever.

I apparently had set the APM to a really low value using hdparm and didn't think it actually applied to ssd's.
Alas it did, by actually lowering the speeds that could be attained on ssd's.

This obviously explains why I could get perfectly fine speeds outside of Arch as well.

Setting APM to 255 (off), I can now max out both of the ssd's

Last edited by Amplificator (2017-09-10 16:05:55)

Arch Linux

#1 2017-09-09 17:02:39

[SOLVED] SSD write speeds seems to be limited to 30 MB/s

#2 2017-09-10 16:04:09

Re: [SOLVED] SSD write speeds seems to be limited to 30 MB/s

Board footer