O/S Slugglish When Moving Files

leokent · 2009-04-20 19:19:26

I am running an updated version of Arch (stock kernel) with ext4 partitions (formatted as ext4) on an M1330 (2.1gz, 3gb ram, 250gb sata hdd) with openbox. If I move or copy large files (1GB each) to different places on the partition using thunar the whole system becomes very sluggish. Most applications seem to become pretty unresponsive in fact. I checked CPU usage and it was only 30% (of one core I believe) and the system load was about 3.5. I do not have a swap partition but I did seem to have memory free. Once the copy/move has finished the system becomes normal again.

Should this be expected or is something funny happening?

[EDIT] Just seen this: http://bbs.archlinux.org/viewtopic.php?id=70525

Regards,
Leo Kent

Last edited by leokent (2009-04-20 19:23:03)

adamlau · 2009-04-20 19:26:36

Not here. My system specs out slower than yours, albeit with XFS partitions. What bogs my system down is:

1. pacman -Syu
2. Creation/extraction of large archives
3. Concatenating large multimedia files
4. Certain sites with Flash64

Last edited by adamlau (2009-04-20 22:32:18)

graysky · 2009-04-28 20:43:20

http://bbs.archlinux.org/viewtopic.php?pid=544678

broch · 2009-04-29 03:45:30

http://bbs.archlinux.org/viewtopic.php?pid=544678

constant reference to swappiness is so wrong that it is not even funny.

Not here. My system specs out slower than yours, albeit with XFS partitions. What bogs my system down is:
1. pacman -Syu
2. Creation/extraction of large archives
3. Concatenating large multimedia files

I have xfs (custom format flags) none of the above affects OS. No idea about Flash64. In other words lots depends on your hardware and configuration.

Wilco · 2009-04-29 07:44:30

I think this is the next big thing to tackle. Dual core is great but a desktop/workstation comes to a halt when copying multiple gigs from one partition to another.

There's nothing you can do about it. If other people say they don't have that problem... they're lying!

iBertus · 2009-04-29 12:42:51

Wilco wrote:

I think this is the next big thing to tackle. Dual core is great but a desktop/workstation comes to a halt when copying multiple gigs from one partition to another.
There's nothing you can do about it. If other people say they don't have that problem... they're lying!

This seems to be an issue that has shown up a few kernel release cycles ago. I used to NOT have this problem when copying large amounts of data, but now it happens everytime I copy GIS data between partitions or drives. Also, it doesn't happen on Windows Vista on either my laptop or my desktop, so I'm not sure you can say that this is something that is just going to happen. It is a recent (last year or so) issue.

edit: I just noticed that the linked thread has referene to this being a problem for ~3-4 years. I only started having this problem within the past year.

edit2: oh, and this is happening on my 8-core Xeon machine w/ 8GB RAM and a 6-drive RAID configuration. Obviously a bug.

Last edited by iBertus (2009-04-29 12:48:22)

lucke · 2009-04-29 12:56:49

Have you guys tried running ionice on the copying process?

I've been alternating between the stock kernel and -zen (with customized config) and oftentimes it seems that -zen performs better - could simply be the issue of something different set in the config.

There's also this: http://lwn.net/Articles/328363/

I never had a desktop stall totally when copying data - it just is a tad less responsive.

Last edited by lucke (2009-04-29 12:57:55)

.:B:. · 2009-04-29 13:18:47

Wilco wrote:

There's nothing you can do about it. If other people say they don't have that problem... they're lying!

I didn't know you had a monopoly on the truth Wilco... Who are you to say other people lie when they do not experience any problems?

Wilco · 2009-04-29 13:44:13

No I'm just really exaggerating, no offense meant

My results in startup time, normal system vs. heavy copying system:
firefox: 4s vs. 15s
urxvt: instant vs. 2-5 s
openoffice.org: 7s vs. 30s

.:B:. · 2009-04-29 13:55:29

I know there have been a few topics around on Linux performance when copying large amounts of data; it even got the /. frontpage once (but I think that was only related at first sight... We all know how /. is).

broch · 2009-04-29 15:03:31

downloaded Fedora 10 dvd iso (3.4GB)
instead of copying, I moved file between partitions:
FF default cold start: ~15s
FF cold start while moving: ~20s
OO.org cold start: ~17s
OO.cold start while moving: 22s
Konqueror cold start: ~4s (slow/I have some widgets added that affect Konqueror response but I like these)
Konqueror cold start while moving: ~8s

I do not use any preload/cache for linux as these are rather primitive when compared to anything windows.
There are few well known issues:
1) hardware compatibility (drivers/bugs)
2) config
3) generally slow file systems when compared to NTFS
4) I/O and kernel (not great)

With all the above I do not see, real slowdown never stalls. I am lucky I guess. I general I consider linux as slow OS (desktop/workstation).

swappiness has in 90% nothing to do with the issue, I would simply leave this as default setting. Any VM manipulation requires a lot of testing to get it tuned properly.

Last edited by broch (2009-04-29 15:04:59)

andre.ramaciotti · 2009-04-29 15:19:00

If I understand the problem correctly, some slowdown while moving/copying is expected. The real problem is that even after the copying was completed there's still a slowdown that shouldn't happen.

broch · 2009-04-29 17:01:23

andre.ramaciotti wrote:

If I understand the problem correctly, some slowdown while moving/copying is expected. The real problem is that even after the copying was completed there's still a slowdown that shouldn't happen.

if this is the case then I do not notice any slowdown at all. This however definitely points to a bug either in kernel or driver.

My first impression was that during copying/moving of large files system slowdown is more pronounced that it should be again this is something that I have not noticed.

Wilco · 2009-04-29 19:11:00

Maybe it is possible to limit the amount of cache a "cp" or "mv" operation can use? And have cp and mv default to a very low priority task which interrupts immediately when some other app wants to access the harddisk?

app4des · 2009-04-30 11:18:53

You can't measure the system load in these situations with CPU% usage., because it is not the CPU that does the copying, but the motherboard's chipset, that usually reaches its limits.

Some chipsets have very weak controllers and when doing much work on Sata/Ide controllers, USB controllers or even ethernet controllers the system slows down to very unresponsive situations. The only way to prevent those things is to limit the speed of the file transfers (I don't know how, it would be useful if someone could post a how-to here).

From my experience high-end motherboards usually use high-end chipsets. For example my cheapish 690G/SB600 motherboard becomes unusable on high speed transfers, however my 790GX/SB750 never does, and both are using the same memory speed and CPU.

If you absolutely want faster response times in all load situations you will have to buy a better motherboard.

Last edited by app4des (2009-04-30 11:20:00)

ngoonee · 2009-04-30 14:21:23

Ah, app4des' explanation seems quite well-informed to me. Always wondered why my Arch64 laptop didn't really experience ANY slowdown when my Arch64 desktop (with a better processor and HD speed) did.

Never ever seen a workaround for this, though. Nobody can really agree where the fault is, in the first place, and until ownership is taken of the problem, its not going to get solved.

Wilco · 2009-04-30 17:46:58

If app4des is right this can't be solved because this is a hardware limitation

loudtiger · 2009-04-30 17:50:51

i don't know if that makes sense. are you saying that some chipsets offload the task to the processor? or that the chipset is the core cause of the slowdown?

ngoonee · 2009-05-01 00:47:30

I believe he's saying that some chipsets do better and some worse. The processor only calculates, basically, other parts of the board are responsible for other things. I guess the chipset handles HD access?

Peterix · 2009-05-02 23:45:14

There's more to this. For example working with any USB storage is damn slow here. Things that should take minutes actually take HOURS and the computer is almost unresponsive while I copy anything bigger than 700MB. Also, the bigger the file, the slower it gets.

60KB/s on a USB2 stick is laughable. This isn't a hardware problem as the same hardware performs well under Windows. It's a kernel problem and a very bad one. First started happening to me around 2.6.18. Stuff got better over time, but never as good as before.
Maybe it's related to locking, because it's all I/O wait. It would make sense - as the devs remove the BKL, things go out of sync and actually slow down instead of getting faster. Anyway, this is just a guess. ~_~

I've got a high-end system. With a damn Intel SSD at its core, a modern mobo/processor and 8G of ram. It shouldn't be this slow.

Just found something: http://news.debuntu.org/content/10488-h … o-wait-bug

darthaxul · 2009-05-03 01:13:51

There is also the fact that your block size is affecting performance, wikipedia says that the larger your block size is, the better performace is with bigger files, as it has to read one big sector instead of many little reads from many sectors. I dont think the filesystem type really matters compared to the block size issue.

jack.mitchell · 2009-05-03 10:59:36

I also have horrendous transfer speeds when putting anything onto USB, the system grinds to a halt and 50% of the time it throws an error or the files don't get copied over correctly. I can see where the suggestion comes from with the motherboard chipset but to me it feels like there is something fundamentally wrong with the copying procedure. It also happens when I have to verify large torrents that I have downloaded, for example I have 10 distro's seeding at 700mb, I select them all and verify the data after my laptops run out of battery and powered off by mistake, this kills the system and takes AGES. Anyway I know this isn't a contribution to the solution but it seems as though it is fairly widespread.

Jack.

Last edited by jack.mitchell (2009-05-03 11:00:30)

Berticus · 2009-05-03 13:37:06

what filesystems are you guys using? I copied a 4 GB file through usb, and the PC was still very usable, and it was done in a few minutes.

app4des · 2009-05-03 21:31:09

For USB transfers make sure you don't use journaling file systems. Journaling FS on USB storage devices don't work good. That and what darthaxul said (the block size, the bigger the better), because USB works in pretty low frequency (many packets per second hog the usb controller).

broch · 2009-05-04 12:36:14

That and what darthaxul said (the block size, the bigger the better), because USB works in pretty low frequency (many packets per second hog the usb controller).

no, block size should be optimized for average file size

Arch Linux

#1 2009-04-20 19:19:26

O/S Slugglish When Moving Files

#2 2009-04-20 19:26:36

Re: O/S Slugglish When Moving Files

#3 2009-04-28 20:43:20

Re: O/S Slugglish When Moving Files

#4 2009-04-29 03:45:30

Re: O/S Slugglish When Moving Files

#5 2009-04-29 07:44:30

Re: O/S Slugglish When Moving Files

#6 2009-04-29 12:42:51

Re: O/S Slugglish When Moving Files

#7 2009-04-29 12:56:49

Re: O/S Slugglish When Moving Files

#8 2009-04-29 13:18:47

Re: O/S Slugglish When Moving Files

#9 2009-04-29 13:44:13

Re: O/S Slugglish When Moving Files

#10 2009-04-29 13:55:29

Re: O/S Slugglish When Moving Files

#11 2009-04-29 15:03:31

Re: O/S Slugglish When Moving Files

#12 2009-04-29 15:19:00

Re: O/S Slugglish When Moving Files

#13 2009-04-29 17:01:23

Re: O/S Slugglish When Moving Files

#14 2009-04-29 19:11:00

Re: O/S Slugglish When Moving Files

#15 2009-04-30 11:18:53

Re: O/S Slugglish When Moving Files

#16 2009-04-30 14:21:23

Re: O/S Slugglish When Moving Files

#17 2009-04-30 17:46:58

Re: O/S Slugglish When Moving Files

#18 2009-04-30 17:50:51

Re: O/S Slugglish When Moving Files

#19 2009-05-01 00:47:30

Re: O/S Slugglish When Moving Files

#20 2009-05-02 23:45:14

Re: O/S Slugglish When Moving Files

#21 2009-05-03 01:13:51

Re: O/S Slugglish When Moving Files

#22 2009-05-03 10:59:36

Re: O/S Slugglish When Moving Files

#23 2009-05-03 13:37:06

Re: O/S Slugglish When Moving Files

#24 2009-05-03 21:31:09

Re: O/S Slugglish When Moving Files

#25 2009-05-04 12:36:14

Re: O/S Slugglish When Moving Files

Board footer