anyone noticed how copying GBs of data makes any linux distro crawl

Army · 2009-02-16 15:38:49

Oh, seems like I missed this thread... I have this issue as well, my system goes to its knees, when for example I copy data from my internal jfs-formatted notebook harddrive to my xfs formatted external harddrive. The CPU load is really low, around 10% or so, so that doesn't seem to be the culprit, but it could happen to take several minutes to simply close an application!
I have a single core CPU with 1,7GHz

Last edited by Army (2009-02-16 15:39:25)

toxygen · 2009-02-17 16:20:30

jordi wrote:

Maybe you guys have the issue described here

wow thanks for the link. i had read another guide related to swappiness, and they suggested setting at 10. setting at 1 massively improved the performance on my core2quad. it's amazing. thanks!

aquila_deus · 2009-02-17 16:40:13

SyXbiT wrote:

I'm pretty sure it's cause of the Ingo Molnar's Completely Fair Scheduler. I don't remember this happening before, but can't be sure

If i copy 50GB of files from one SATA 7200rpm ext3 drive to another, all apps are excruciatingly slow (including firefox, nautilus etc..) until the copying is finished
whereas non-linux kernels (vista, OS X etc..) don't have this issues.
In vista, firefox seems pretty responsive even while copying.
has anyone else noticed?

Is your SATA disk recognized as ATA or SCSI in linux? Such a problem usually happens when ATA hd is in PIO mode, in which the DMA is disabled and disk access consumes a lot of CPU resource.

Generally XFS should offer the best performance when nobarrier option is used, but the choice of filesystem and IO scheduler should be unrelated to your problem I believe.

broch · 2009-02-17 17:32:27

with xfs use deadline I/O scheduler
optimize formating flags:

for example
mkfs.xfs -l version=1,internal,size=128m -i size=2048 -d agcount=2 -f /dev/sdX

xfs likes big log files, does not lime barriers (which nowadays is not a problem)

I/O scheduler should be selected for specific fs and tasks.

there is interesting disscussion at lkml about constant (useless?) tweaking of I/O scheduler code vs file system

Ranguvar · 2009-02-17 20:25:17

A patch that may help: http://bugzilla.kernel.org/attachment.cgi?id=20172

That, the swappiness setting, and the cache pressure setting, should help. I found an older patch from Con Kolivas to autoregulate the swappiness setting, but it doesn't patch on recent kernels and I don't have the skills to make it work (or faith that it is still relevant).

Original message: http://lkml.org/lkml/2003/10/23/53
Fixed patch: http://lkml.org/lkml/2003/10/25/4

I also found this: http://ck.kolivas.org/patches/swap-pref … h-41.patch
With explanation here: http://lwn.net/Articles/153353/

broch · 2009-02-17 21:05:46

Ranguvar wrote:

A patch that may help: http://bugzilla.kernel.org/attachment.cgi?id=20172
That, the swappiness setting, and the cache pressure setting, should help. I found an older patch from Con Kolivas to autoregulate the swappiness setting, but it doesn't patch on recent kernels and I don't have the skills to make it work (or faith that it is still relevant).
Original message: http://lkml.org/lkml/2003/10/23/53
Fixed patch: http://lkml.org/lkml/2003/10/25/4
I also found this: http://ck.kolivas.org/patches/swap-pref … h-41.patch
With explanation here: http://lwn.net/Articles/153353/

taken from the patch refernce page ( http://bugzilla.kernel.org/show_bug.cgi?id=12309 ):

Does not seem to fix the latency problem though.

Ranguvar · 2009-02-17 21:32:50

Yes. It does mitigate the damage, though.

http://bugzilla.kernel.org/show_bug.cgi?id=12309#c166

enito · 2009-03-05 15:04:07

Hi, any tip for fix this? i move from a sata to another using mc , and the machine turn really slow....any more tips is really welcomed..

Rehto · 2009-04-18 08:09:22

I already said this in another thread but I'll say it again in this one
I managed to get decent performance with the anticipatory scheduler
(used as default in kernel <=2.6.18 - enable it by 'echo anticipatory > /sys/block/$devicename/queue/scheduler'
as root and add 'elevator=as' to your menu.lst) and adding noatime to fstab...

Last edited by Rehto (2009-04-18 08:18:50)

Procyon · 2009-04-18 11:00:49

Rehto wrote:

I already said this in another thread but I'll say it again in this one
I managed to get decent performance with the anticipatory scheduler
(used as default in kernel <=2.6.18 - enable it by 'echo anticipatory > /sys/block/$devicename/queue/scheduler'
as root and add 'elevator=as' to your menu.lst) and adding noatime to fstab...

Thanks, I will definitely try that out, and by <=2.6.18 do you mean >=2.6.18?

Rehto · 2009-04-18 11:26:04

I meant 2.6.18 and older kernels

Procyon · 2009-04-18 11:43:04

I have .19 and it does say anticipatory in the scheduler file.
EDIT:
$ cat scheduler
noop anticipatory deadline [cfq]

Last edited by Procyon (2009-04-18 11:44:08)

Ranguvar · 2009-05-21 20:57:55

The first few words there are schedulers you _can_ use. The one in brackets is the one you _are_ using.

smarcher · 2009-05-22 14:26:19

I haven't notice this much on Arch, but at work I have Ubuntu 9.04 ext4, the comp almost slows down even while copying from one directory to another on the same hard disk. Seems to be more pronounced in ext4.

Ranguvar · 2009-08-07 01:57:54

A patch _some_ are saying helps, should apply to 2.6.31-rc5 and possibly lower. If anyone wants to try..? I cannot right now.

http://bugzilla.kernel.org/attachment.cgi?id=22180

xstaticxgpx · 2009-08-07 09:49:37

Ranguvar wrote:

A patch _some_ are saying helps, should apply to 2.6.31-rc5 and possibly lower. If anyone wants to try..? I cannot right now.
http://bugzilla.kernel.org/attachment.cgi?id=22180

I applied the patch to -rc5 and booted up with cfq and tried

dd if=/dev/zero of=/tmp/test bs=1M count=1M

and it was just as bad and definitely seemed worse than my previous straight deadline setup.
I use ext4 on all my desktop HDD's

R00KIE · 2009-08-07 11:04:34

I have seen somewhere (can't remember where now) that for people with sata HDs the deadline scheduler would help a bit, that's what I am using now and I don't see my pc crawling to a halt but it is possible to notice a certain slowdown.

I'm running with 4G ram, no swap an no extra patches.

graysky · 2009-08-07 19:11:33

http://bbs.archlinux.org/viewtopic.php?id=71131 <--- another thread describing this problem

Last edited by graysky (2009-08-07 19:43:48)

Zariel · 2009-08-07 19:22:37

Im using zen patch set and CFQ and dont have this problem anymore (or as bad as it was)

Beini · 2009-09-03 12:04:23

I'm also affected by this, and I now really had a chance to test some changes

I started doing a partition conversion on a 500gb partition with convertfs and it pretty much froze the system and it took like 2 hours just issue these two things (the conversion is still going while I'm writing this so at least the desktop is usable now):

1 . Change to deadline scheduler

echo deadline > /sys/block/sda/queue/scheduler

echo deadline > /sys/block/sdb/queue/scheduler

2. Start ioReniceD ( http://bbs.archlinux.org/viewtopic.php?id=73359 )

The difference is huge. From unusable to slight lagginess

EDIT: Oops, I'm not sure does ioreniced require cfq. But I actually started it before changing the scheduler

Last edited by Beini (2009-09-03 12:07:11)

Sophotect · 2009-09-03 16:36:42

http://lwn.net/Articles/350100/

shining · 2009-09-03 17:08:40

Sophotect wrote:

http://lwn.net/Articles/350100/

This is a CPU scheduler though, does it really fix the I/O problems ?

I seem to remember Con was also working on I/O interactivity at some point, but I was not yet able to find back any materials.

Edit : he is reporting himself the very same problem in http://ck.kolivas.org/german_linux_maga … erview.txt :

Disk I/O is a disaster for the desktop. Most other hardware gets faster
exponentially, yet hard disks are still based on a 50 year old design and are
only getting faster and fatter at a linear rate. This means they are becoming
_relatively_ slower and slower. Anything we do that can avoid and minimise
disk I/O will help. Write a very large file to (one) disk and see what else
you can do on your desktop while this is going on.

Last edited by shining (2009-09-03 17:16:54)

ngoonee · 2009-09-03 22:59:30

shining wrote:

Sophotect wrote:
http://lwn.net/Articles/350100/
This is a CPU scheduler though, does it really fix the I/O problems ?
I seem to remember Con was also working on I/O interactivity at some point, but I was not yet able to find back any materials.
Edit : he is reporting himself the very same problem in http://ck.kolivas.org/german_linux_maga … erview.txt :
Disk I/O is a disaster for the desktop. Most other hardware gets faster
exponentially, yet hard disks are still based on a 50 year old design and are
only getting faster and fatter at a linear rate. This means they are becoming
_relatively_ slower and slower. Anything we do that can avoid and minimise
disk I/O will help. Write a very large file to (one) disk and see what else
you can do on your desktop while this is going on.

It's an IO scheduler, not a CPU scheduler.

Ranguvar · 2009-09-03 23:34:29

Hell yeah!! Can't wait to try this out.

And Zariel, the Zen patchset includes the patch I mentioned actually

shining · 2009-09-04 05:59:51

ngoonee wrote:

shining wrote:
Sophotect wrote:
http://lwn.net/Articles/350100/
This is a CPU scheduler though, does it really fix the I/O problems ?
I seem to remember Con was also working on I/O interactivity at some point, but I was not yet able to find back any materials.
Edit : he is reporting himself the very same problem in http://ck.kolivas.org/german_linux_maga … erview.txt :
Disk I/O is a disaster for the desktop. Most other hardware gets faster
exponentially, yet hard disks are still based on a 50 year old design and are
only getting faster and fatter at a linear rate. This means they are becoming
_relatively_ slower and slower. Anything we do that can avoid and minimise
disk I/O will help. Write a very large file to (one) disk and see what else
you can do on your desktop while this is going on.
It's an IO scheduler, not a CPU scheduler.

How can you be so confident in writing non-sense ?

Arch Linux

#26 2009-02-16 15:38:49

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#27 2009-02-17 16:20:30

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#28 2009-02-17 16:40:13

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#29 2009-02-17 17:32:27

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#30 2009-02-17 20:25:17

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#31 2009-02-17 21:05:46

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#32 2009-02-17 21:32:50

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#33 2009-03-05 15:04:07

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#34 2009-04-18 08:09:22

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#35 2009-04-18 11:00:49

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#36 2009-04-18 11:26:04

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#37 2009-04-18 11:43:04

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#38 2009-05-21 20:57:55

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#39 2009-05-22 14:26:19

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#40 2009-08-07 01:57:54

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#41 2009-08-07 09:49:37

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#42 2009-08-07 11:04:34

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#43 2009-08-07 19:11:33

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#44 2009-08-07 19:22:37

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#45 2009-09-03 12:04:23

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#46 2009-09-03 16:36:42

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#47 2009-09-03 17:08:40

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#48 2009-09-03 22:59:30

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#49 2009-09-03 23:34:29

Re: anyone noticed how copying GBs of data makes any linux distro crawl

#50 2009-09-04 05:59:51

Re: anyone noticed how copying GBs of data makes any linux distro crawl

Board footer