You are not logged in.

#1 2010-12-24 02:34:24

Clouseau
Member
Registered: 2010-12-24
Posts: 112

USB flash drive 5times faster with ntfs filesystem?

Hello,

I have a weird problem with my USB Voyager GT 16GB flash drive.  I formated it in ext2/ext3/xfs/fat32 and then start copying a file >1GB in size. The speed of transfer very quickly drops to 2-4MB/s. That is very slow. And if I format it in ntfs it works as it should be 16-17MB/s in linux with ntfs-3g. I really don't get it. I've been playing with all sort of options all day and is driving me nuts big_smile Tried even another kernel with BFS and BFQ I/O sheduler kernel26-ck and the problem is identical. Tried turning of legacy support in BIOS for USB and nothing, the problem is still here. Mount it it with "async" option which gave me speed of 800KB/s... Is it possible that Voyager GT 16GB doesn't work good with any other filesystem? Or maybe I'm missing something... Clearly it isn't problem with usb 2.0 ports because all of them are 2.0 and the ntfs speed wouldn't be that good if the flash was working in usb 1.1  mode. Any thought would be much welcome smile

Thanks in advance,

Clouseau

Last edited by Clouseau (2010-12-24 02:42:06)

Offline

#2 2010-12-24 08:42:13

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: USB flash drive 5times faster with ntfs filesystem?

What were the mount options you used?

Offline

#3 2010-12-24 09:17:59

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: USB flash drive 5times faster with ntfs filesystem?

Odd, I did the experiment on my Kingston Technology DataTraveler 112 4GB Pen Drive. (is this USB2.0 I can't tell).

$ cd /dev/shm
$ dd if=/dev/zero of=testfile bs=1M count=1024 conv=fdatasync,notrunc
<<made new partition of whatever format and manually mounted it>>
$ rsync -P /dev/shm/testfile /media/test

ext2:noatime:2.3 MB/s
ntfs:ntfs-3g:5.5 MB/s

EDIT:  http://www.kingston.com/flash/dt112.asp?id=2 - Specs are 10MB/sec. read and 5MB/sec. write so I guess the ntfs numbers are fine.  Why are the ext2 numbers half of the ntfs numbers?

Last edited by graysky (2010-12-24 09:20:42)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#4 2010-12-24 09:41:08

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

karol wrote:

What were the mount options you used?

First I've mounted it through Dolphin and the mount options were:
ext2 -  /dev/sdc1 on /media/disk type ext2 (rw,nosuid,nodev,uhelper=hal) - speed is 2-4MB/s
ntfs - /dev/sdc1 on /media/disk type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions) - spped is good - 15-16MB/s


and then i've do that in terminal:
ext2 - /dev/sdc1 on /media/USB type ext2 (rw) - these were default options, no options were given to mount command. . I've also tried with sync:   /dev/sdc1 on /media/USB type ext2 (rw,noexec,nosuid,nodev,sync,noatime) and the speed is even worse - around 650KB/s
ntfs -  /dev/sdc1 on /media/USB type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096) - these were default options, no options were given to ntfs-3g command. the speed is good - 16-17MB/s

I've also tried with another USB flash which is 2GB and the results are identically disproportional.

In win7 it also works at good speed - ntfs - 16MB/s.

Really, don't get it...

graysky wrote:

Odd, I did the experiment on my Kingston Technology DataTraveler 112 4GB Pen Drive. (is this USB2.0 I can't tell).

$ cd /dev/shm
$ dd if=/dev/zero of=testfile bs=1M count=1024 conv=fdatasync,notrunc
<<made new partition of whatever format and manually mounted it>>
$ rsync -P /dev/shm/testfile /media/test

ext2:noatime:2.3 MB/s
ntfs:ntfs-3g:5.5 MB/s

Try the drive with different ports, maybe some aren't 2.0 and try it in Windows with ntfs to see if the speed has encreased.

EDIT: i've seen that you added the link to speed specs of your drive. So at ntfs it reaches full speed. What are your mount options?

Last edited by Clouseau (2010-12-24 09:44:50)

Offline

#5 2010-12-24 09:50:34

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: USB flash drive 5times faster with ntfs filesystem?

@cluoseau - they're in my original post which you quoted smile


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#6 2010-12-24 10:25:56

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

graysky wrote:

@cluoseau - they're in my original post which you quoted smile

Ah, i've slept for 4h so maybe that is the reason I didn't see them big_smile

Last edited by Clouseau (2010-12-24 10:26:42)

Offline

#7 2010-12-24 15:19:24

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: USB flash drive 5times faster with ntfs filesystem?

Did you explicitly specify in the mount options that you wanted the partition mounted asynchronously (for the ext*/fat* filesystems)?


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#8 2010-12-24 16:04:18

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

R00KIE wrote:

Did you explicitly specify in the mount options that you wanted the partition mounted asynchronously (for the ext*/fat* filesystems)?

Yes, that was in the options which gave me speeds of 2-4MB/s. If I specify sync for synchronously the speed drops to 650KB/s which is very bad.

I have also tried this with the Kubuntu 10.10 boot live cd and the behaviour is identical. The motherboard is Asrock P55 Extreme4 and maybe the reason for this kind of behaviour is that ehci controller in kernel doesn't work very good with usb drives and my hardware... I cannot think of any other reason. The pc is rather new so maybe the latest kernel doesn't fully support my motherboard?

But sata ext4 to anothet sata ext4 works very speedy, copying is at 65-70MB/s...

EDIT: It appears the problem is present for a longer time across various distros:
https://bugs.launchpad.net/ubuntu/+sour … bug/541937
https://bugs.launchpad.net/ubuntu/+sour … bug/197762

UPDATE: I've tested it with noop, deadline, cfq and bfs scheduler, the result is the same...

Last edited by Clouseau (2010-12-24 17:50:36)

Offline

#9 2011-09-07 12:14:14

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

I'm seriously thinkg to toss computer out throught the windows cause this problem is driving me insance. I'm loosing hours every day on constat reformatting of my USB drive. In windows no problem, but in linux, it drives me crazy! Slow! Slow! Slow! Its the same on another computer... The problem must be in schedulers, i've tried them all. I'm now on the latest kernel and it's still driving me nuts...

USB Hard disk have lesser problems than this stick... But in windows everything is fine with the stick...

And also when I copy all system slows down, it becomes unresponsive...

I'm on x64 Arch with KDE4, and my system is up to date. The problem is here for couple of years and I'm thinking really to switch back to windows cause I cannot use my computer for basic things... I mean I do not need it for Adobe Premier Pro, but I would like to transfer couple of mkv files now and then I don't want to loose two hours every time I wan't to do that... Damn, I'm mad sad

This is my cat /proc/interrupts. The rescheduling interrupts are very high.

   CPU0       CPU1       CPU2       CPU3       
   0:        348          0          0          0   IO-APIC-edge      timer
   1:      82420          0          0          0   IO-APIC-edge      i8042
   8:          1          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          4          0          0          0   IO-APIC-edge      i8042
  16:     768552          0          0          0   IO-APIC-fasteoi   mei, ehci_hcd:usb1
  17:        171          0          0          0   IO-APIC-fasteoi   hda_intel
  18:          0          0          0          0   IO-APIC-fasteoi   ata_generic
  23:   16242602          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2
  40:          0          0          0          0   PCI-MSI-edge      PCIe PME
  41:          0          0          0          0   PCI-MSI-edge      PCIe PME
  42:          0          0          0          0   PCI-MSI-edge      PCIe PME
  43:          0          0          0          0   PCI-MSI-edge      PCIe PME
  44:          0          0          0          0   PCI-MSI-edge      PCIe PME
  45:    4636694          0          0          0   PCI-MSI-edge      ahci
  46:   65645608          0          0          0   PCI-MSI-edge      eth0
  47:    5047842          0          0          0   PCI-MSI-edge      hda_intel
  48:    6234037          0          0          0   PCI-MSI-edge      nvidia
NMI:      10780       7413       3864       2774   Non-maskable interrupts
LOC:   28950146   27310427   27977434   15954004   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:      10780       7413       3864       2774   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:   27411973   44067980   37191358   46027973   Rescheduling interrupts
CAL:     236457     357518     433390     486080   Function call interrupts
TLB:     410269     343394     159266     107912   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:        303        303        303        303   Machine check polls

Offline

#10 2011-09-07 12:19:21

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

Bumping on a desk with my head...

Last edited by Clouseau (2011-09-07 12:22:10)

Offline

#11 2011-09-07 12:20:07

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,490

Re: USB flash drive 5times faster with ntfs filesystem?

Some filesystem implement a cache. The kernek does not write the data immediatly to the disk and this give the illusion that it is faster. than it is actually. sync will syncronize the data and the disk. To see the real speed do

sync ; time sh -c '<command that create a big file> ; sync'

Offline

#12 2011-09-07 12:24:50

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

No, the problem is somewhere else. If I reboot the sytem the copying is fast. nest day I plug the stick, it drops from 16MB/s to 200KB/s. I now it isn't fake becase it took me 5min for 5GB when it works. But when it starts to behave stupid then 5GB lasts for two hours... and my computer is inresponsive... Don't get it, really.

https://bbs.archlinux.org/viewtopic.php?pid=981013

Offline

#13 2011-09-07 14:33:16

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: USB flash drive 5times faster with ntfs filesystem?

Maybe you need USB in mkinitcpio hooks?


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#14 2011-09-07 17:06:19

thisoldman
Member
From: Pittsburgh
Registered: 2009-04-25
Posts: 1,172

Re: USB flash drive 5times faster with ntfs filesystem?

The slow transfer to a USB device is such a persistent problem for so many users, I wish we could find the source of it.  I don't suffer from it.

Transfers of large files to vfat-formatted USB flash drives are on the order of 8 to 16 MB/s for me.  It's a desktop PC, 4GB RAM, hard-wired ethernet connection.  The printer and a USB hub on the keyboard are the only constant USB connections.  The keyboard itself and the mouse use USB to PS/2 connectors.

I never use suspend or hibernate.  The machine is powered off and cold-booted twice a day.  Could this be the difference?

I mount USB sticks manually, they are listed in fstab.  The effective options are:

$ mount
...
/dev/sdi1 on /media/hita type vfat (rw,noexec,nosuid,nodev,noatime,shortname=lower,flush,utf8,gid=100)

Other possible differences:

$ uname -a
Linux chickadee 3.0-ARCH #1 SMP PREEMPT Tue Aug 30 08:53:25 CEST 2011 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux

$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 004: ID 058f:6377 Alcor Micro Corp. Multimedia Card Reader

$ lsmod    ### Edited clip shown
sd_mod                 28307  12 
pata_amd               11395  0 
pata_acpi               3376  0 
ohci_hcd               21714  0 
sata_nv                21812  9 
libata                173297  3 pata_amd,pata_acpi,sata_nv
scsi_mod              131546  6 usb_storage,uas,sg,sr_mod,sd_mod,libata
ehci_hcd               39511  0 
usbcore               142576  5 usb_storage,uas,ohci_hcd,ehci_hcd

$ ls /etc/modprobe.d
modprobe.conf           pcspkr_blacklist.conf       usb-load-ehci-first.conf
nouveau_blacklist.conf  snd_layla24_blacklist.conf  usblp_blacklist.conf

$ grep -inT usb /etc/mkinitcpio.conf
  49   :#    HOOKS="base udev pata scsi sata usb filesystems"
  55   :##   This setup loads an lvm2 volume group on a usb device.
  56   :#    HOOKS="base udev usb lvm2 filesystems"
  57   :HOOKS="base udev autodetect pata scsi sata filesystems usbinput"

Does anything, beside the frequent bootups, seem significant?

Last edited by thisoldman (2011-09-07 17:11:35)

Offline

#15 2011-09-07 19:48:04

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: USB flash drive 5times faster with ntfs filesystem?

Some mobos do not have sufficient current to support 2.0 flash drives, perhaps you can measure the current provided when running in a good speed and compare it with what happens when it doesn't run at good speed.

Otherwise, it sounds like the device is being reset from ehci to uhci by some quirk in the system.  Perhaps even power management reduces the current supplied.


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#16 2011-09-07 20:57:22

defears
Member
Registered: 2010-07-26
Posts: 218

Re: USB flash drive 5times faster with ntfs filesystem?

Try BS=4096, just like ntfs-3g did. The normal default is 512, I think.

Offline

#17 2011-09-08 10:52:30

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

The computer is HP 8200 Elite CMT in translation its quad i5, it has 4gb of RAM and to disks of 500GB in stripe RAID so this should fly very good but when I try to usb copy, it laggs like a penitum 4 with 256MB ram and XP on it with antivirus software installed... I'm google it to find any answers now and then, but I'll I find is that peple have this problem for years and is being ignored... Frustrating like hell... I have tried today to copy again to usb, and the speed is about 8MB which is usually dobule when it works. I started firefox adn it drops to 600KB, firefox is runnig by ps -ef but is not shoing the gui. If I unplug the usb stick forcely, Firefox GUI comes instant so it must have be something with the shedulers. I have also tries noop and deadline, the results are similar...

Offline

#18 2011-09-08 20:53:34

defears
Member
Registered: 2010-07-26
Posts: 218

Re: USB flash drive 5times faster with ntfs filesystem?

I don't have a computer that exhibits this problem now, (used to) but could you disable cpufreq or powernow and make sure you are not being throttled by frequency scaling. I just noticed that I have not had that problem for 2 years and I ditched cpufreq about 2 years ago. Coincidence, idk.

Offline

#19 2011-09-12 10:50:24

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

My powerplan is performance only. I have now disabled acpi_cpureq module from cpufrequtils utils and the problem persists. I have put 10GB to copy on USB stick, I turned on firefox and started browsing, opening tabs etc. and the speed drops from 15MB/s to 2MB/s... And my FF becomes unresponsive and the whole computer lags...

Offline

#20 2011-09-12 14:32:20

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: USB flash drive 5times faster with ntfs filesystem?

What size cpu cache in your mobo?


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#21 2011-09-12 19:12:27

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: USB flash drive 5times faster with ntfs filesystem?

lilsirecho wrote:

What size cpu cache in your mobo?

Not to be sarcastic, but that was core i5, so L2 size is probably 512Kb -- 1Mb and L3 is either big or BIG.

@Clouseau:
I agree, that having a journalled FS perform much better that non-journalled is wierd... Are you saying that data transfer with ntfs-3g is unaffected by system load, while ext[2-4] is affected?

I think you should open a bugreport somewhere (I know kernel.org is down, but in the nearest future).


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#22 2011-09-12 19:27:12

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: USB flash drive 5times faster with ntfs filesystem?

Many people are experiencing slow transfer, high I/O etc., there are many threads on this forum e.g. https://bbs.archlinux.org/viewtopic.php?id=126261 about slow copying to exFAT.
No solution and not everyone is affected. NTFS as a "solution" however sounds ... interesting :-)

Leonid.I wrote:

I think you should open a bugreport somewhere (I know kernel.org is down, but in the nearest future).

A little bit OT: http://www.phoronix.com/scan.php?page=n … &px=OTYwMA

Offline

#23 2011-09-13 08:04:12

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

Yes, my theory is correct... The schedulers are bad. I have now tried again deadline and noop (don't have anticipatory in kernel) and the results are:
1) CFQ system stalls and becomes unresponsive, I can only move mouse. If I force unplug the usb stick, instant revival of the whole system and is again pretty fast
2) DEADLINE everything is very slugish while copying, the system doesn't stall for a longer period of time, I can move through windows but very slowly and sluggish
3) NOOP - this one behaves better than the two before but still not even close to perfect. I can open other windows and scrool through them, thunderbird works ok but if I do something in firefox, open few tabs and close, FF becomes unresponsive... This comment was written while copying and while writting the line 3) I couldn't write anymore, FF was blocked and the speed rate of copying to usb dropped to 500KB/s...

With Noop its a little better but can't understand why the FF is so heavily blocked while other applications are just sluggish...

EDIT: tried to format usb to ext2... same inresponsivnes, but the speed is all the time below 1MB/s... So I switched back to ntfs for usb stick and I did "mkfs.ntfs /dev/sdd1" and it starded. When it was at 3% all system stalled, I coudn't even move a mouse, nothinh nadda... So I had to unplug the USB stick form the slot and instat reviaval and responsivnes of the whole system... Gone again 2h of my life with "playing" with usb sticks today... If I smash the computer with my IBM M keyboard who would win: keyboard or the computer?

Last edited by Clouseau (2011-09-13 08:23:33)

Offline

#24 2011-09-13 08:25:24

Clouseau
Member
Registered: 2010-12-24
Posts: 112

Re: USB flash drive 5times faster with ntfs filesystem?

Leonid.I wrote:
lilsirecho wrote:

What size cpu cache in your mobo?

Not to be sarcastic, but that was core i5, so L2 size is probably 512Kb -- 1Mb and L3 is either big or BIG.

@Clouseau:
I agree, that having a journalled FS perform much better that non-journalled is wierd... Are you saying that data transfer with ntfs-3g is unaffected by system load, while ext[2-4] is affected?

I think you should open a bugreport somewhere (I know kernel.org is down, but in the nearest future).


Naaah, they don't listen to much... I have opened before and nothing, complete ignore, not even a response...

Offline

#25 2011-09-14 04:29:07

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: USB flash drive 5times faster with ntfs filesystem?

Clouseau wrote:

<snip> If I smash the computer with my IBM M keyboard who would win: keyboard or the computer?

Depends what sort of casing/monitor you have.

Clouseau wrote:
Leonid.I wrote:

I think you should open a bugreport somewhere (I know kernel.org is down, but in the nearest future).

Naaah, they don't listen to much... I have opened before and nothing, complete ignore, not even a response...

[mod-note] You know you can combine posts or edit previous posts? Please don't double-post where possible.

And in my experience kernel bugs only draw attention when they're actually useful and not simply "this is slow when I do this".


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

Board footer

Powered by FluxBB