You are not logged in.

#1 2019-06-18 13:34:28

casted
Member
Registered: 2019-06-18
Posts: 2

iowait spikes when writing files - encrypted SSD

First post, so hello world.

I've been running arch on this hardware for about a year now, with no problems at all.

Hardware is a thinkpad x230 with an i5-332M and sandisk 240GB SSD (2015). Detailed info on the hardware.

I've recently updated my setup, switching to full disk encryption (LVM on LUKS).

The install is recent so the SSD is about a quarter full. Discard/TRIM is not enabled.

Ever since, whenever writing files on my disk the SSD writing speed slows down to a crawl and iowait goes up to 70-90%, rendering the system unusable.

It doesn't seem the CPU is ramping up that much.

cryptsetup bechmark outputs:

aes-xts        512b       957.8 MiB/s       983.1 MiB/s

I understand it's a RAM test but it should take the blame away from the CPU and towards the SSD.

This was first noticed when copying my data from a backup.
I have replicated the problem with dd.
When the count is low, there is no freeze and the writing speeds clock at about 160MB/s (writing a single 1G block).
When writing multiple large-ish files (100M to 1G anywhere between 8 and 50, larger blocks need lower replicates) speed goes down to about 20MB/s and (which is what bothers me) the system becomes unusable.

I'm surely missing something and am happy to run some other tests, but I'm not sure which direction I should investigate. Here are some monitors:

They are about mid process (I have the whole monitoring, I can pastebin it it can help but don't think it is particularly interesting, it's basically flat to these values for both tests).
What I ran on this particular occasion was:

dd if=/dev/zero of=test bs=1G count=8 status=progress

mpstat

02:25:51 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:28:30 PM  all    1.20    0.00    4.27   88.74    0.25    0.27    0.00    0.00    0.00    5.27

iostat

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.33    0.00    3.82   93.72    0.00    1.13

Beyond hardware failure it might concievably be:

  1. Me setting up encryption wrong

  2. Discard/TRIM (write amplification?)

  3. some kernel setting I should edit

Any ideas/suggestions are very welcome, I am more than ready to re-format and re-install if that's what it takes but I would like to do it knowing what I did wrong (I already tried).

Thanks in advance!

Offline

#2 2019-06-18 14:24:19

sabroad
Member
Registered: 2015-05-24
Posts: 242

Re: iowait spikes when writing files - encrypted SSD

Firstly, check partition alignment as this is important for SSD write amplification.

Some SSD devices without DRAM may perform badly without enabling dmcrypt Discard/TRIM support so this can be tried.

For large writes, the default multiqueue scheduler can end up filling multiple queues of sequential IO that look like random IO (to some devices that have trouble with internal multiqueue scheduling), so it may be worth trying the "none" queuing algorithm to see if this improves things.

Failing that, it may be worth benchmarking without LUKS (but with LVM) to confirm or rule-out dmcrypt-related issues on your setup.

Last edited by sabroad (2019-06-18 15:05:47)


--
saint_abroad

Offline

#3 2019-06-20 20:50:21

casted
Member
Registered: 2019-06-18
Posts: 2

Re: iowait spikes when writing files - encrypted SSD

Thanks for the great suggestions, sorry it took some time to try them all.

The partitions were aligned. I've enabled TRIM and implemented the "none" queuing algorithm.

Things have slightly improved. especially with the queuing algorithm, but the problem persists. I haven't benchmarked without LUKS yet (and my previous install on the same disk was MBR, so it honestly might be that).
On this  post (linked from the wiki) I found that:

Encrypted data can't be compressed. If your SSD controller uses compression internally to minimize writes and maximize performance (SandForce controllers do this) you may notice a performance hit.

My controller is, indeed SandForce. Does anyone by any chance know if it can be disabled?

I believe I will now try to:

  • reset disk with blkdiscard (never done it, plus I might have zeroed it out once in the past)

  • check if LVM (unencrypted) gives the same results

  • eventually proceed to install unencrypted arch and use a veracrypt container for my personal stuff (my threat model is some guy stealing my laptop)

Last edited by casted (2019-06-21 06:04:10)

Offline

Board footer

Powered by FluxBB