USB3 sticks: slow write speed

ezacaria · 2016-01-02 10:56:33

Hi,

I have some issues writing to my USB3 sticks. Write performance is really slow, and I suspect that this was brought by a recent system upgrade - I do not have the numbers before the upgrade, but I would remember if the sticks were THAT slow

Unfortunately, I wiped the pacman cache recently and cannot downgrade easily. Anyways, it would be interesting to see if we can identify the bottleneck in the current versions.

Using up to date system, current kernel is 4.3.3-2-ARCH.

Measurement methodology: clock the execution of cp for a 3.3GB file, followed by sync via "time" in the shell (I wrapped the cp and sync in a bash script, then call time to execute it).

The source file is in tmpfs, and there is RAM to spare. The machine has no load otherwise, therefore this seems a good estimate of the real overall throughput.

First, I clocked the USB3 port against other devices, to make sure it is not the USB3 controller or the connector. The issue seems to affect only the USB sticks so far.

Tested two USB3 external hard disks, speed for the complete 3.3GB file transfer (+sync) is somewhere between 55-80 MB/s.
3.5 inch "green" drive on external drive dock over eSATA to USB3 converter: 80 MB/s (nasty performance degradation introduced by the adapter, same drive does 100 MB/s over pure eSATA)
SSD inside USB3 enclosure: 115 MB/s. Definitely low considering the capabilities of the SSD disk inside the enclosure, but good enough for now.
As reference, 3.5inch 7200 rpm drive on a removable disk tray built into the tower achieves 161 MB/s.

Now for the sticks. I chose the brand because it is performing very well on all machines where I have tested them.
Until recently, they were writing decently in the Linux desktop (always somewhat slower compared to using them in other machines, but "decent enough") and now something has changed.

First, read performance is still OK. The system can read files of 3-4 GB size to tmpfs at 170-180 MB/s.

Write performance, however, has turned into a disaster. I am currently getting 20-30 MB/s on the USB3 port, and 10-20 MB/s on the USB2 port.

I am sure that the sticks can perform much better:

Same machine booting windows 7: there is no equivalent measuring commands, but based on rough clocking of copy + "eject removable" operation, write speed should be > 100 MB/s.
Same stick on a windows 8 laptop. Reported speed is above 120 MB/s and time to eject the drive is negligible, therefore it seems that speed shown by w8 is close to true. Moreover, I copied back the file to the Linux box and contents seemed to be OK.

After reading some articles on the net, I thought that the system may be pushing too much data to the sticks at the time.
Therefore, I made some experiments by changing the max_sectors for the drive ("echo N > /sys/block/sd.../device/max_sectors").
The only clear conclusion to me is that using too low values (less than 100, e.g., 64) starts affecting performance.
While the sticks or drives do not crash if the value is pushed up (e.g., up to 960 from default 240), there is no clear throughput improvement either.
The amount of "Writeback" that can be observed in /proc/meminfo changes with max_sectors, but there seems to be no net effect throughput-wise.

The controller is from ASMedia and built-in on the motherboard: lspci reports "ASM1042 SuperSpeed USB Host Controller", driver is xhci_hcd and module xhci_pci

Looking at lsusb for the stick and the external drive, it seems that the driver in use is different.

One can read online that "The idVendor and idProduct are used by the operating system to find a driver for your device", and this is clearly different in my case:

The stick:

Bus 008 Device 013: ID 8564:1000 Transcend Information, Inc. JetFlash
  idVendor           0x8564 Transcend Information, Inc.
  idProduct          0x1000 JetFlash

whereas the drive uses the "ASM1051 SATA bridge":

Bus 008 Device 014: ID 174c:5106 ASMedia Technology Inc. ASM1051 SATA 3Gb/s bridge
  idVendor           0x174c ASMedia Technology Inc.
  idProduct          0x5106 ASM1051 SATA 3Gb/s bridge

Now, there was bug 84611 reported roughly a year ago "USB 3.0 stick only running at USB 2.0 speed" against "3.14-stable and newer (including 3.16.2)"

The bug report mentions that kernel 3.14-rc6 was not affected, but I do not find any mentioning of kernel 4.x. Besides, the fact that the stick reads at USB3 speed suggests that this may be something new.

I think that before trying with an older or different kernel, it would make sense to try to use an alternative driver, if any.
Or perhaps it would be possible to downgrade the driver without changing the kernel.

On a side note, the 0x13fe Kingston drivers are also giving very bad performance, but I think this was always so

Any ideas?

Thanks in advance!

Eduardo

tom.ty89 · 2016-01-02 11:00:59

Well what's the filesystem on it though? It could be just slow fuse/ntfs-3g...

ezacaria · 2016-01-02 11:38:08

Sorry, I forgot to mention it. But the external drives are using NTFS, so ntfs-3g is not the bottleneck.

R00KIE · 2016-01-02 12:32:28

I'd say tom.ty89 is on to something, for the sake of getting it out of the way, test with a native filesystem such as ext4. Also make sure to test with large files as many small files kill the speed.

Alternatively you could test how fast dd can write to the "raw" device, that should avoid any of the pitfalls of testing with a filesystem. Do search the forum on how to use dd to do that test.

ezacaria · 2016-01-02 13:31:53

Well, there may be something about the combination of filesystem and driver.

As I wrote earlier, NTFS-3g is not a bottleneck when writing to hard disk-like devices (using the ASM1051 driver in the case of the external USB3 drive, or whatever is used with one of the internal desktop drives, which is NTFS formatted).

However, the combo of NTFS+ 0x1000 (JetFlash) driver is now problematic. This seems to me a recent issue, but I do not have hard data to prove that the performance was better before.

Anyways, and back to the suggestion. I formatted the stick as ext4 and retested with the 3.3 GB filee. The initial copy after formatting yielded only 63 MB/s, but subsequent operations showed 120-125 MB/s. Similar performance (minus the initial hiccups) was obtained after formatting to xfs and jfs.

But, then I formatted the other stick with ext4 as well (the one using the Kingston driver), and the performance is still around 30 MB/s.

While the improvement of ext4/xfs/jfs + jetFlash driver is is good news to some extent, it is not a solution because I need to use the stick in windows machines

I also tested the JetFlash stick with exfat filesystem, and it had bad performance. I do not know if we can blame slow implementation on that

Anyways, do you think there could be some way to tune the ntfs parameters to improve the writing speed?

Thanks!

R00KIE · 2016-01-03 01:44:05

In my experience ntfs-3g has always been slow and a cpu hog. That said you can try the big_writes mount option, if that doesn't work you have to wait for someone else to chime in or try to ask the ntfs-3g guys directly in their support forum [1]. Maybe you can find some tips or similar problems there.

[1] http://tuxera.com/forum/

ezacaria · 2016-01-04 20:18:56

Pity about your woes with ntfs-3g. My own experience has been far better across the years.

About the ntfs-3g performance: I tested writing the 3.3 GB file to two different SSDs. Over the USB3 port, I reached 115-125 MB/s speeds, and around 140 MB/s with the big_writes option. This improved to about 150-160 MB/s when writing a larger file (5.8 GB). Considering that XFS/EXT4 performance on the same drive and link is about 170 MB/s, I would say ntfs-3g is pretty competitive here - more so because of the interoperability advantage. CPU load was below 10% on the core executing the copy.

And to abstract from the USB3 link itself, I clocked the transfer to an SSD connected via internal SATA, resulting in 302 and 150 MB/s with and without the big_writes option, respectively. This is pretty good in my eyes, considering that using ext4 the transfer goes at 383 MB/s.

As to my original problem: the best and most consistent performance for the stick so far seems to be using XFS. But since I do need the NTFS, I will have to keep digging. The big_writes option did not help anything with the stick, unlike the case with the SSDs.

Soukyuu · 2016-01-05 21:06:13

You might also want to test exfat, on my own tests it performs better than ntfs, even though I'm using fuse-exfat module.

alphaniner · 2016-01-05 22:24:41

ezacaria wrote:

Looking at lsusb for the stick and the external drive, it seems that the driver in use is different.
One can read online that "The idVendor and idProduct are used by the operating system to find a driver for your device", and this is clearly different in my case:
The stick:
Bus 008 Device 013: ID 8564:1000 Transcend Information, Inc. JetFlash
  idVendor           0x8564 Transcend Information, Inc.
  idProduct          0x1000 JetFlash
whereas the drive uses the "ASM1051 SATA bridge":
Bus 008 Device 014: ID 174c:5106 ASMedia Technology Inc. ASM1051 SATA 3Gb/s bridge
  idVendor           0x174c ASMedia Technology Inc.
  idProduct          0x5106 ASM1051 SATA 3Gb/s bridge

For ther record, this is normal. Your stick was branded and/or manufactured by Transcend and is part of the JetFlash product line. OTOH, your enclosure is "powered by" an ASMedia chipset which translates between USB and SATA.

tom.ty89 · 2016-01-06 06:10:40

You mean uas versus usb-storage? That doesn't really have much to do with brands. It's just whether the chip supports UASP or not. It is getting common in SATA-USB adapters, but general thumb drive still rarely support that. FWIW UASP was supposed to make things faster too:
https://en.wikipedia.org/wiki/USB_Attached_SCSI

[tom@localhost ~]$ lsusb.py -ui
...
usb2             1d6b:0003 09  3.00 5000MBit/s 0mA 1IF  (Linux 4.3.3-2-ARCH xhci-hcd xHCI Host Controller 0000:00:14.0) hub
 2-1             174c:55aa 00  3.00 5000MBit/s 0mA 1IF  (asmedia ASMT1051 123456789415)
  2-1:1.0         (IF) 08:06:62 4EPs (Mass Storage:SCSI) uas host0 (sda)
 2-4             0781:5583 00  3.00 5000MBit/s 896mA 1IF  (SanDisk Ultra Fit 4C531001370212105524)
  2-4:1.0         (IF) 08:06:50 2EPs (Mass Storage:SCSI:Bulk-Only) usb-storage host7 (sdc)
...
[tom@localhost ~]$

ezacaria · 2016-01-06 08:18:32

I just lost the answer I had previously composed because the login time experired, so bear with me if I am a bit telegraphic on this one

I traced the copy operations via iostat, and then made some statistics with python+pandas.

My interpretation is that the filesystem driver is generating write requests that end up being merged at the block layer, and the resulting write requests issued to the device are being processed slowly.

While it is true that "better" write requests could be generated (as XFS does, please refer to the table below showing that there's little merging in case of XFS), the slow handling of the "merged" requests could still be a problem in the IO device driver. In my eyes, this is still no proof that the filesystem is at fault.

Anyways, please find the results below. All values are averages per test (a 3.3GB file copied on each test). In the table, "bw" or "big_w" denotes usage of "big_writes" ntfs-3g mount option, and all rows refer to the stick, except where "_ssd" indicates the SSD over USB3 is in use.
I also computed an estimation of the write sector size in the "sectorSize_calc" column (in bytes), as throughput / (number of write requests * size of the request in sectors), which suggests that all cases used 512 byte sectors when writing to the device (independent of the fact that 512B sector emulation may be in use by the drives)

              wMB/s  avgrq-sz  avgqu-sz  w_await    wrqm/s      w/s  sectorSize_calc
NTFS_big_w    30.48    238.01    289.50  1404.96   7539.69   261.99           512.47
NTFS          41.15    238.10    266.24  1239.02  10180.90   352.93           513.47
EXFAT         29.39    237.63    285.96  1342.13   7271.46   252.49           513.65
NTFS_bw_ssd  149.62    238.51    254.97   197.45  37017.64  1284.91           511.94
XFS          117.73    227.95    123.66   182.03     44.04  1059.44           511.15
XFS_SSD      149.62    226.77    128.43    91.82      1.82  1346.59           513.77

tom.ty89 · 2016-01-06 13:39:36

FWIW, can you tell exactly which stick do you have? http://www.transcend-info.com/product/usb

I can only see one stick from Transcend that marks a write speed of 140MB/s (close to the performance you claimed you experienced in Windows): http://www.transcend-info.com/Products/No-378
So is this the stick you have?

All other sticks marks 45MB/s or below. This is the second fastest: http://www.transcend-info.com/Products/No-505

ezacaria · 2016-01-06 13:50:12

Yes, that is the one (on the "No-378" page, 64GB capacity).

tom.ty89 · 2016-01-06 14:43:20

Can you run the following commands as root/with sudo and paste the output here?

smartctl -d scsi -x /dev/sdX
smartctl -d sat --get wcache /dev/sdX

where /dev/sdX is the stick.

ezacaria · 2016-01-06 15:03:01

Here they are:

smartctl -d scsi -x /dev/sdk
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.3.3-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               JetFlash
Product:              Transcend 64GB
Revision:             1.00
Compliance:           SPC-4
User Capacity:        63,245,910,016 bytes [63.2 GB]
Logical block size:   512 bytes
Rotation Rate:        Solid State Device
Serial number:        2411669617
Device type:          disk
Local Time is:        Wed Jan  6 15:49:20 2016 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Disabled
Temperature Warning:  Disabled or Not Supported
Read Cache is:        Disabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Error Counter logging not supported

Device does not support Self Test logging
Device does not support Background scan results logging

and

smartctl -d sat --get wcache -T permissive /dev/sdk
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.3.3-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error unsupported scsi opcode

Write cache is:   Unavailable

Thanks!

AndyD · 2016-05-01 11:00:04

I've got very similar problems with my external drive, a "Seagate Expansion Portable Drive" with 1TB memory.

The read performance is fine (especially with cp, over 100MB/s), but write performance is only around 13MB/s. It does not make a difference if the drive is formatted to ntfs or to ext4, whereas Win 7 on the same machine has no problems writing with 100MB/s (obviously only on ntfs).

I'm using an up-to-date x86_64-linux-4.5.1-1-ARCH kernel, and the USB chipset is Intel, not ASMedia (I cut USB messages for other controllers throughout these snippets):

$ lspci -knn | grep USB -A3
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
	Subsystem: CLEVO/KAPOK Computer Device [1558:2300]
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

$ journalctl -b | grep usb
May 01 12:24:02 kernel: usb 2-2: new SuperSpeed USB device number 2 using xhci_hcd
May 01 12:24:02 mtp-probe[1779]: checking bus 2, device 2: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-2"
May 01 12:24:02 kernel: usbcore: registered new interface driver usb-storage
May 01 12:24:02 kernel: usbcore: registered new interface driver uas

$ lsusb
Bus 004 Device 002: ID 8087:8000 Intel Corp. 
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 8087:8008 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 0bc2:231a Seagate RSS LLC 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 5986:055c Acer, Inc 
Bus 001 Device 002: ID 8087:07dc Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The relevant fstab entry, once for ntfs and once for ext4:

#/dev/sdc1 UUID="40c8a488-057d-4d74-9bf4-31ae1d7aab0e" LABEL="Virtual Machine"
LABEL=VM /mnt/vm ntfs user,noauto,rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other  0 0
#LABEL=VM /mnt/vm ext4 rw,relatime,data=ordered,user,exec  0 0

And the smartctl commands:

$ sudo smartctl -d scsi -x /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               Seagate
Product:              Expansion
Revision:             0706
Compliance:           SPC-4
User Capacity:        1,000,204,885,504 bytes [1.00 TB]
Logical block size:   512 bytes
LB provisioning type: unreported, LBPME=-1, LBPRZ=0
Logical Unit id:      0x3e41383241313830
Serial number:        NA82A180
Device type:          disk
Local Time is:        Sun May  1 12:48:38 2016 CEST
SMART support is:     Unavailable - device lacks SMART capability.
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===

Error Counter logging not supported

No self-tests have been logged

Device does not support Background scan results logging

$ sudo smartctl -d sat --get wcache /dev/sdc

Write cache is:   Enabled

Any help solving this would be greatly appreciated! Since the drive is quite new, and I might be able to return it, alternative tips on USB 3 drives that actually work out of the box with arch at full speed would also be nice

ezacaria · 2016-05-01 15:16:37

In the meanwhile, I am mostly using the stick to read data in Linux. When writing from Linux I am using a portable USB3 drive, which is NTFS-formatted and able to write somewhere >90MB/s. Or I am using the stick when I am not in a hurry

I had some bad experience when writing from Linux to a 1GB Seagate drive as well. What I can recommend based on my experience is the USB3 Transcend portable drives.

For 1GB, I recommend the StoreJet M3 (anti-shock version) - if you can still find it. For 2GB, the StoreJet H3P is also fine.

Arch Linux

#1 2016-01-02 10:56:33

USB3 sticks: slow write speed

#2 2016-01-02 11:00:59

Re: USB3 sticks: slow write speed

#3 2016-01-02 11:38:08

Re: USB3 sticks: slow write speed

#4 2016-01-02 12:32:28

Re: USB3 sticks: slow write speed

#5 2016-01-02 13:31:53

Re: USB3 sticks: slow write speed

#6 2016-01-03 01:44:05

Re: USB3 sticks: slow write speed

#7 2016-01-04 20:18:56

Re: USB3 sticks: slow write speed

#8 2016-01-05 21:06:13

Re: USB3 sticks: slow write speed

#9 2016-01-05 22:24:41

Re: USB3 sticks: slow write speed

#10 2016-01-06 06:10:40

Re: USB3 sticks: slow write speed

#11 2016-01-06 08:18:32

Re: USB3 sticks: slow write speed

#12 2016-01-06 13:39:36

Re: USB3 sticks: slow write speed

#13 2016-01-06 13:50:12

Re: USB3 sticks: slow write speed

#14 2016-01-06 14:43:20

Re: USB3 sticks: slow write speed

#15 2016-01-06 15:03:01

Re: USB3 sticks: slow write speed

#16 2016-05-01 11:00:04

Re: USB3 sticks: slow write speed

#17 2016-05-01 15:16:37

Re: USB3 sticks: slow write speed

Board footer