Very slow speed when when writing to SSD-drive

johannesWinter · 2014-04-08 13:18:16

Hi everyone,

I recently changed to Arch Linux, before I used Fedora. I followed the Installation Guide from the Wiki. I varied from the standard setup by choosing to encrypt my root filesystem as was described there.
I have troube with the speed of my SSD however, it's very slow:

dd if=/dev/zero of=./somefile bs=1024 count=5242880            
^C1578798+0 records in
1578798+0 records out
1616689152 bytes (1.6 GB) copied, 682.503 s, 2.4 MB/s

dd if=/dev/zero of=./somefile bs=1024 count=5242880  0.33s user 7.50s system 1% cpu 11:22.50 total

I aborted the command, since it took so long, but one can see it only manages 2.4MB/s. I didn't have this problem when using Fedora, but I also didn't use encryption. I am aware that encryption slow down the drive, but I don't think it shoud be that much slower.

I have enabled trim by adding it too my bootloader and to fstab:

LABEL arch
    MENU LABEL Arch Linux
    LINUX ../vmlinuz-linux
    APPEND cryptdevice=/dev/sda2:cryptroot:allow-discards root=/dev/mapper/cryptroot rw 
    INITRD ../initramfs-linux.img

cat /etc/fstab 
# 
# /etc/fstab: static file system information
#
# <file system>	<dir>	<type>	<options>	<dump>	<pass>
# /dev/mapper/cryptroot
UUID=b1e2846a-b800-4a4a-a055-171d3d0ad771	/         	ext4      	rw,relatime,discard,data=ordered	0 1

# /dev/sda1
UUID=939d9f1b-1efc-4fab-a8c0-6ae4737e563f	/boot     	ext4      	rw,relatime,discard,data=ordered	0 2

# restrict tmpfs size?
tmpfs   /tmp            tmpfs   defaults,noatime,mode=1777,size=15%     0 0
tmpfs   /var/tmp        tmpfs   defaults,noatime,mode=1777,size=5%      0 0

However I'm not sure if trim is working, following this howto I don't think it is: http://andyduffell.com/techblog/?p=852
Could the slow speed be attributed to trim not working?

I also have an external usb drive, which is encrypted with truecrypt. I can write to it with high speeds, the problem is only with my build in SSD.

Info about my encrypted partition and the drive:

sudo cryptsetup status cryptroot 
/dev/mapper/cryptroot is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  device:  /dev/sda2
  offset:  4096 sectors
  size:    117020431 sectors
  mode:    read/write
  flags:   discards

sudo hdparm -i /dev/sda                                                                                           :(

/dev/sda:

 Model=OCZ-VERTEX2, FwRev=1.29, SerialNo=OCZ-53KCK9S9258ODAIR
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=117231408
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4 
 DMA modes:  mdma0 mdma1 mdma2 
 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: unknown:  ATA/ATAPI-2,3,4,5,6,7

 * signifies the current active mode

I think my aes-support is working:

lsmod | grep aes
aesni_intel            45644  6 
aes_x86_64              7399  1 aesni_intel
lrw                     3565  1 aesni_intel
glue_helper             4449  1 aesni_intel
ablk_helper             2004  1 aesni_intel
cryptd                  8409  4 ghash_clmulni_intel,aesni_intel,ablk_helper

I just tried to post the output from dmesg with nopaste, but I only get:

dmesg | nopaste       
http://pastie.org/pastes

I don't find it there then. I'll add it soon.

Hope anybody has an idea where my problem lies. Let me know what other information I should supply.

Thanks in advance

Johannes

frostschutz · 2014-04-08 14:06:25

bs=1024 is too small, try bs=1M or bs=64k or something

with the default bs=512 or similar small bs=1024 you're measuring CPU overhead

you should also check your partition alignment

# parted /dev/sda unit s print
Model: ATA M4-CT064M4SSD2 (scsi)
Disk /dev/sda: 125045424s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start  End         Size        File system  Name     Flags
 1      2048s  125044735s  125042688s               primary  lvm

if start sector is a multiple of 2048 you should be fine for any medium; if it's something odd like 63 you'll have to start over

for cryptsetup, try 'cryptsetup benchmark'. with aes-ni active, it should be fast for aes-xts, depending on your cpu. my i5-4570 goes at >2000MB/s, and it works fine for my SSD.

Last edited by frostschutz (2014-04-08 14:06:36)

johannesWinter · 2014-04-08 14:27:44

I put the output from dmesg here:
http://pastie.org/private/cprjtnsu24q72fkkjuisjq

I noticed some errors like this:

[ 1474.695990] ata1.00: status: { DRDY }
[ 1474.695993] ata1.00: failed command: WRITE FPDMA QUEUED
[ 1474.695999] ata1.00: cmd 61/00:30:08:0e:39/04:00:06:00:00/40 tag 6 ncq 524288 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

I had already fixed this problem by adding libata.force=noncq to my kernel parameters, I guess I forgot it when I added allow-discards

So here is the dmesg after with libata.force=noncq added:
http://pastie.org/private/c5gkanfihqsunbf6sdgyva

I noticed something else. I created several 1G files using dd. The speed slowed down with each file:

dd if=/dev/zero of=./swapfile2 bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 65.5381 s, 16.4 MB/s
dd if=/dev/zero of=./swapfile2 bs=1024 count=1048576  0.27s user 5.58s system 8% cpu 1:05.54 total

dd if=/dev/zero of=./swapfile3 bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 144.525 s, 7.4 MB/s
dd if=/dev/zero of=./swapfile3 bs=1024 count=1048576  0.25s user 6.08s system 4% cpu 2:24.53 total

dd if=/dev/zero of=./swapfile4 bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 356.969 s, 3.0 MB/s
dd if=/dev/zero of=./swapfile4 bs=1024 count=1048576  0.23s user 5.23s system 1% cpu 5:56.97 total

My guess is that the first file was written to space which was formely occupied by another big file. When I deleted the former file trim worked it's magic hopefully. This made me remember, when I followed the Installation guide there was a step I did which overwrote my drive with random data.
What I'll try now is creating a file that takes up all remaing space like above and then deleting it. Do you believe this might work?

johannesWinter · 2014-04-08 14:36:39

About the partition alignment, it's a multiple of 2048:

parted /dev/sda unit s print
Model: ATA OCZ-VERTEX2 (scsi)
Disk /dev/sda: 117231408s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start    End         Size        File system  Name              Flags
 1      2048s    206847s     204800s     ext4         Linux filesystem  legacy_boot
 2      206848s  117231374s  117024527s               Linux filesystem

Benchmark:

cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       832203 iterations per second
PBKDF2-sha256     557753 iterations per second
PBKDF2-sha512     388937 iterations per second
PBKDF2-ripemd160  546133 iterations per second
PBKDF2-whirlpool  172463 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   636.5 MiB/s  1563.9 MiB/s
 serpent-cbc   128b    60.1 MiB/s   255.9 MiB/s
 twofish-cbc   128b   146.0 MiB/s   203.4 MiB/s
     aes-cbc   256b   488.2 MiB/s  1228.0 MiB/s
 serpent-cbc   256b    63.2 MiB/s   253.8 MiB/s
 twofish-cbc   256b   151.0 MiB/s   208.0 MiB/s
     aes-xts   256b  1366.7 MiB/s  1382.2 MiB/s
 serpent-xts   256b   225.2 MiB/s   233.2 MiB/s
 twofish-xts   256b   192.9 MiB/s   192.5 MiB/s
     aes-xts   512b  1112.1 MiB/s  1121.6 MiB/s
 serpent-xts   512b   227.0 MiB/s   231.3 MiB/s
 twofish-xts   512b   191.7 MiB/s   192.7 MiB/s
cryptsetup benchmark  4.49s user 23.76s system 99% cpu 28.489 total

johannesWinter · 2014-04-08 15:03:23

I created another file with dd as suggested. This failed :-(. My root was remounted read-only.

dd if=/dev/zero of=./swapfile6 bs=1M count=2048
dd: error writing ‘./swapfile6’: Read-only file system
509+0 records in
508+0 records out
533585920 bytes (534 MB) copied, 503.914 s, 1.1 MB/s

The according dmesg output looks pretty bad to me ...
http://pastie.org/private/t1dagb2wy5awoir9rwaynw

Arch Linux

#1 2014-04-08 13:18:16

Very slow speed when when writing to SSD-drive

#2 2014-04-08 14:06:25

Re: Very slow speed when when writing to SSD-drive

#3 2014-04-08 14:27:44

Re: Very slow speed when when writing to SSD-drive

#4 2014-04-08 14:36:39

Re: Very slow speed when when writing to SSD-drive

#5 2014-04-08 15:03:23

Re: Very slow speed when when writing to SSD-drive

Board footer