[SOLVED] SSD wear leveling and slow performance ...

jjb2016 · 2017-05-16 15:05:40

Hi All,

I have a Samsung 840 Pro SSD 128GB, which has been used in a ZFS pool for a few months now. It has a small partition which is mirrored with another device partition for the ZFS SLOG and another partition with the remainer of the space for the ZFS L2ARC. I've been experience very slow performance issues when doing certain tasks with data in the zpool, especially when working within the virtual machines who's images reside on the zpool. I used "iostat -dx 5" to monitor the IO of all of the physical devices and noticed a 100% utilization of this SSD, quite a lot of the time. I then carried out a read benchmark using gnome-disks and found the average read rate to be around 140MB/s when it should be around 550MB/s. Looking at the SMART Data report I'm seeing a wear-leveling-count value of 1118.

Here is the full smartctl output:

smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.10.13-1-custom] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 840 PRO Series
Serial Number:    S1ANNSAF322455P
LU WWN Device Id: 5 002538 5a02001cd
Firmware Version: DXM05B0Q
User Capacity:    128,035,676,160 bytes [128 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May 16 16:16:18 2017 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (65476) seconds.
Offline data collection
capabilities:                    (0x53) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  15) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       3727
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       38
177 Wear_Leveling_Count     0x0013   069   069   000    Pre-fail  Always       -       1118
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   060   046   000    Old_age   Always       -       40
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       33
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       13118141529

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3726         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
  255        0    65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Am I right in thinking that this SSD is on its last legs? I have another identical SSD in the system, which is used for the main OS (Arch of course) and a small partition mirror with the other SSD for the ZFS SLOG. The wear-leveling-count value for that drive is 472. I don't fully understand the different columns for the report. I'm seeing Value, Normalised, Threshold, Worst, Types, Updates, Assessment ... an explanation of these would be welcome. The values I've mentioned here are from the Value column.

Thanks.

Last edited by jjb2016 (2017-05-18 16:28:54)

Slithery · 2017-05-16 15:13:02

I think the issue here is using one device for both the L2ARC and the SLOG, this is not recommended and will likely give you worse performance than not using it at all.

jjb2016 · 2017-05-16 15:21:57

Thanks for the comment slithery. I could try removing the SLOG and L2ARC from the zpool and monitor performance. Do you have any comment on the wear-leveling-count value?

jjb2016 · 2017-05-16 15:27:46

Just add to the info on my zpool config, in case it help identify why this SSD appears to be dying:

pool: zfsdata
 state: ONLINE
  scan: scrub repaired 0 in 4h26m with 0 errors on Mon May  8 03:18:49 2017
config:

        NAME                                            STATE     READ WRITE CKSUM
        zfsdata                                         ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            ata-WDC_WD2002FAEX-007BA0_WD-WMAWP0386996   ONLINE       0     0     0
            ata-WDC_WD2002FAEX-007BA0_WD-WMAY05143895   ONLINE       0     0     0
            ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC16019CFAA  ONLINE       0     0     0
          raidz1-1                                      ONLINE       0     0     0
            ata-WDC_WD2503ABYZ-011FA0_WD-WMAYP0DP2KMZ   ONLINE       0     0     0
            ata-WDC_WD2503ABYZ-011FA0_WD-WMAYP8350900   ONLINE       0     0     0
            ata-WDC_WD5003ABYX-01WERA2_WD-WMAYP3368224  ONLINE       0     0     0
        logs
          mirror-2                                      ONLINE       0     0     0
            slog                                        ONLINE       0     0     0 ####### this is a 2GB partition on /dev/sdf
            slog                                        ONLINE       0     0     0 ####### this is a 2GB partition on /dev/sdh
        cache
          l2arc                                         ONLINE       0     0     0  ####### this is on /dev/sdh

errors: No known data errors

/dev/sdh is the dying SSD here. You can see it's used for the cache (L2ARC) and log (SLOG).

Ropid · 2017-05-16 15:32:17

I didn't quite get your explanations. Does the way you use it mean that the drive is basically 100% full at all times? If yes, try to leave something like 10% or 20% unallocated and see what happens to the performance. After you've shrunk a partition to get those 20% unused space, create a new partition in that spot so that you can run "blkdiscard <devicename>" to send TRIM commands to the drive for that area, then remove the partition again.

jjb2016 · 2017-05-16 15:41:47

No the drive isn't 100% full at all times. It's a cache drive which gradually fills up (takes days / weeks to fill up with the way I'm using it). What I meant before was that the drive utilization appears to be at 100% in terms of IO. I can understand this drive being busy as a ZFS cache device but the 100% utilization at the time of measuring seemed a little odd, and this lead me to do the gnome-disks benchmark - which showed very poor read performance (i only did the read test).

I take your point though about leaving some unallocated space for overprovisioning. This is something that a learned about only after I setup this zpool.

What I'm looking for in this thread is some opinions about the state of this SSD considering the wear-leveling-count value. Does this seem high? If this value isn't particularly high then I'll investigate other causes for the slow performance.

Last edited by jjb2016 (2017-05-16 15:45:50)

R00KIE · 2017-05-16 17:32:57

It might be a firmware problem. Those samsung drives were akin to lemons[1-4].

[1] http://www.anandtech.com/show/8550/sams … on-the-way
[2] http://www.anandtech.com/show/8570/firm … tober-15th
[3] http://www.anandtech.com/show/9158/new- … this-month
[4] http://www.anandtech.com/show/9196/sams … 40-evo-fix

Ropid · 2017-05-16 17:40:30

Those articles are about the 840 EVO (and maybe the previous 840), not about the 840 Pro.

Ropid · 2017-05-16 18:24:34

I tried to research this a bit. I don't quite understand all of what I browsed through, but I got the feeling that something strange is happening for you with that wear leveling count. Besides the raw value that appears to be really high, there's also the column named "value" that's supposed to be the manufacturer's interpretation of what the raw value means. It says 69 for you, and from what I gathered that's supposed to mean it has 69% life left. This does not really mean that the drive is supposed to be close to dead at 0%, it instead just means that the manufacturer would refuse warranty on the argument that you've written too much to it

What's really strange is that you apparently have not written that much data to the drive. According to that "total LBAs written" value you have, you wrote 6.1 TB data. Formula I used is:

13118141529 * 512 / 2^40
 = 6.1

I have two drives here that I compared to yours:

Samsung 850 EVO 512GB: it has 5 TB written and its wear leveling count is just raw value 6, and value 99%.

Crucial MX100 256GB: it has 11.5 TB written, "ave block-erase count" raw value 124, and value 96%.

My best guess would be that it maybe never got sent TRIM commands for any part of that virtual "disk surface" it presents to the PC, and at some point all of the disk was written to, and now the SSD controller has problems. I don't get why this would show in the read speeds. Perhaps things can somehow get fragmented internally and the controller has no room to rewrite stuff so everything is now slow?

mich41 · 2017-05-16 18:27:27

For lifetime estimation you want to look at the normalized value, which is 69 and probably started from 100.

These guys wrote over a petabyte to a few SSDs and didn't find them to slow down with time.

However, I think it normally is the case that SSDs slow down after lots of random writes due to fragmentation in the internal flash remapping layer. This could be improved by more overprovisioning or secure-erasing the drive and putting data back.

R00KIE · 2017-05-17 10:46:27

@Ropid
You are most probably right, the 840 EVO uses TLC while the 840 Pro uses MLC and is most probably not affected by the same bug.

@jjb2016
Have you noticed the speed problem shortly after turning on and booting the system or does it persist after a long time? The reason I'm asking is that attribute 235 (POR_Recovery_Count) seems to indicate that the power to the ssd is being cut of without telling the ssd that power is going to be cut off. This means that the ssd might have to do some recovery work once it gets power again. Do you have the ssd connected to any add-on controller card?

You could also try running the long smart self-test and see if any errors come up or if any smart parameters degrade and then check the speed afterwards. You should also look at the speed graph and not only the average speed result as it might provide clues as to what the problem might be.

jjb2016 · 2017-05-17 17:13:25

Hi All,

I have done some experimenting today .... firstly I've removed the SSD from the zpool and wiped it with zeros using dd. I then used "blkdiscard /dev/sdh" to well ... discard blocks! So now at least I know that performance issues probably have nothing to do with the SSD being part of the zpool - it's isolated. After this I benchmarked the disk using gnome-disks and got a consistent average read of 410MB/s and write of 258MB/s. So this is better than what I was seeing when i started the thread. But still, this is well below what the SSD should be capable of. As I said before I have a second identical SSD in the system, which is used as the main boot device, which measures average read speed of 550MB/s + in the same benchmark.

I wonder if the SATA port that it's plugged into could make a difference? I have a ASUS Sabertooth 990fx R2.0 mobo which has a total of 8 SATA ports but 2 of them are different ... from the manual:

"AMD ® SB950 Chipset
Supports AMD ® QUAD-GPU CrossFireXTM Technology
- 6 x SATA 6.0 Gb/s ports with RAID 0, 1, 5 and 10 support

ASMedia ® SATA 6Gb/s controllers
- 2 x SATA 6Gb/s ports
- 2 x eSATA 6Gb/s ports [red]"

I wonder if the ASMedia ports perform differently to the AMD SB950 ports.

Rookie - the speed appears to be pretty consistent now after reboot.
Regarding power disruptions - none that I'm aware of but perhaps my powersupply needs upgrading. I have a 500W PSU at the moment which is driving 2 x SSD + 6 x HDDs in the system (plus sound card and GPU). But I imagine that if the PSU was struggling then I would have seen other issues as well, but I have not.

@slithery: following your advice I have now removed the slog devices from the zpool, and will continue with just this SSD as a cache device. It's probably safer to not use a separate log until I can afford to buy a couple of dedicated devices for the job. I've read that they should ideally have some kind of power cut protection as well.

Thanks,

Last edited by jjb2016 (2017-05-17 17:14:43)

frostschutz · 2017-05-17 18:25:55

The asmedia can't use the full SATAIII bandwidth for a single port, and it has two ports. It makes no difference for hard disks, and not for SSD either as long as you're more concerned about access times than linear throughput. But if you have the choice it's better to use the asmedia for slow media.

R00KIE · 2017-05-17 21:59:31

I wasn't implying that the problem is with the PSU you are using. I was thinking more along the lines of whatever port/controller where you have the ssd connected is not passing some commands along which causes the ssd to register an unexpected power loss.

It might also be because of this: https://lkml.org/lkml/2017/4/10/1181. Do note that the post does include information about the proper shutdown sequence for SSDs (and I would guess that spinning rust disks need something very similar).

jjb2016 · 2017-05-18 16:28:22

frostschutz wrote:

The asmedia can't use the full SATAIII bandwidth for a single port, and it has two ports. It makes no difference for hard disks, and not for SSD either as long as you're more concerned about access times than linear throughput. But if you have the choice it's better to use the asmedia for slow media.

I'm pretty convinced now that the reduced performance of this SSD now is down to the fact that I've got it connected to one of the ASMedia ports. So right now I have 6 x HDD connected to the 6 x ADM SB960 ports, and the 2 x Samsung SSDs connected to the 2 x ASMedia ports. I would rather maximise the performance of the 6 x HDDs.

I'll probably close the thread now - thanks to everybody who chipped in! I still don't know if wear-levelling-count value of 1118 is high or not but I won't lose sleep over it.

frostschutz · 2017-05-18 21:27:53

Well, try moving two of your HDD to the Asmedia, with any luck it will improve your performance with zero downsides (since HDDs are too slow to give the asmedia any trouble).

Arch Linux

#1 2017-05-16 15:05:40

[SOLVED] SSD wear leveling and slow performance ...

#2 2017-05-16 15:13:02

Re: [SOLVED] SSD wear leveling and slow performance ...

#3 2017-05-16 15:21:57

Re: [SOLVED] SSD wear leveling and slow performance ...

#4 2017-05-16 15:27:46

Re: [SOLVED] SSD wear leveling and slow performance ...

#5 2017-05-16 15:32:17

Re: [SOLVED] SSD wear leveling and slow performance ...

#6 2017-05-16 15:41:47

Re: [SOLVED] SSD wear leveling and slow performance ...

#7 2017-05-16 17:32:57

Re: [SOLVED] SSD wear leveling and slow performance ...

#8 2017-05-16 17:40:30

Re: [SOLVED] SSD wear leveling and slow performance ...

#9 2017-05-16 18:24:34

Re: [SOLVED] SSD wear leveling and slow performance ...

#10 2017-05-16 18:27:27

Re: [SOLVED] SSD wear leveling and slow performance ...

#11 2017-05-17 10:46:27

Re: [SOLVED] SSD wear leveling and slow performance ...

#12 2017-05-17 17:13:25

Re: [SOLVED] SSD wear leveling and slow performance ...

#13 2017-05-17 18:25:55

Re: [SOLVED] SSD wear leveling and slow performance ...

#14 2017-05-17 21:59:31

Re: [SOLVED] SSD wear leveling and slow performance ...

#15 2017-05-18 16:28:22

Re: [SOLVED] SSD wear leveling and slow performance ...

#16 2017-05-18 21:27:53

Re: [SOLVED] SSD wear leveling and slow performance ...

Board footer