Arch Linux

mir91

Banned

Registered: 2013-11-17

Posts: 143

USB hard drive transfer forces system into a cawl

I attached an external USB 2.0 hdd and stored data on it at a rate of about 35MB/s. This took really long, cause it was a larger backup. CPU load didn't exceed 20%.
However, the whole system suddenly because slow to react and there were really bad freeze spikes coming up repeatedly that would completely freeze the system for like 10 seconds or so.
Basically the system because very hard to use, and that over the course of some hours till the backup was finished.
It's a Core i7 with 8 GB RAM of which about 6GB were free.
Now I think that must be a bug in the kernel, since a mere USB 2.0 transfer shouldn't be able to freeze this kind of system?
I tried a workaround by specifying

ionice -c 3 -p <pid of the copy process>

but although to me it seemed that the general system slowdown disappeared, the frequent freezes remained.
Are there maybe any unofficial kernel patches available for USB transfers that would prevent the system from freezing? Any other solution?

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: USB hard drive transfer forces system into a cawl

It's a known issue and it has been present since forever.
What filesystems, what schedulers, are you using the stock Arch kernel? Does it happen with the -ck kernel?

What command did you use exactly to move your data?

Grima

Member

Registered: 2011-09-02

Posts: 58

Re: USB hard drive transfer forces system into a cawl

I feel for you, OP. I’ve been plagued by this since ... I don’t even remember anymore! Maybe 2.5 years? Before that, I could transfer files and be a happy user at the same time. Now though, I’ve come to accept it (“time to copy, time to afk”) since I’ve tried most things I could think of.

For about a year, I’ve been running the -ck kernel; but I can’t say I notice any difference regarding USB transfers—my system still gets stalled by them. Maybe your system won’t...? It’s worth a try.

Well, best of luck! This is a nasty issue.

MSC

Member

Registered: 2014-07-13

Posts: 33

Re: USB hard drive transfer forces system into a cawl

I've seen this bug too, only on USB drives for some strange reason. I think it's caused by evicting too much cached memory at once. Going to /proc/sys/vm and adjusting dirty_ratio, dirty_background_ratio, dirty_writeback_centisecs, and dirty_expire_centisec (specifically, lowering the ratio and increasing the expiration time) seems to help, but it might also hurt performance elsewhere..

mir91

Banned

Registered: 2013-11-17

Posts: 143

Re: USB hard drive transfer forces system into a cawl

It's a known issue and it has been present since forever.
What filesystems, what schedulers, are you using the stock Arch kernel? Does it happen with the -ck kernel?

What command did you use exactly to move your data?

I write into a truecrypt container that's on an NTFS-formatted USB stick and contains an Ext4 file system.
I used two commands, both cause these problems, the first one being even worse than the second one.

truecrypt -t -c <..create encrypted container file..>
dd if=/dev/sda of=/<..truecrypt container mount point> bs=1M

No idea what schedulers, how do I find out?

Stock kernel yes, except I had to patch it with:

CONFIG_HZ_1000=y
CONFIG_HZ=1000

which I needed for my NVidia GPU to prevent it from crashing randomly when Bumblee daemon tries to enable it and enter an unusable state until system gets rebooted.
This goes with kernel parameters:

rcutree.rcu_idle_gp_delay=1 acpi_enforce_resources=lax

.

I haven't tried that "ck" kernel because I read how people reported that it didn't help them with this problem.

gedgon

Member

From: Poland

Registered: 2011-01-27

Posts: 95

Re: USB hard drive transfer forces system into a cawl

Have you guys tried setting Transparent Hugepage to madvise https://wiki.archlinux.org/index.php/Pe … ge_devices
and lowering a virtual memory to, eg.

vm.dirty_background_bytes = 16777216
vm.dirty_bytes = 16777216

https://wiki.archlinux.org/index.php/Sy … ual_memory

mir91

Banned

Registered: 2013-11-17

Posts: 143

Re: USB hard drive transfer forces system into a cawl

Have you guys tried setting Transparent Hugepage to madvise https://wiki.archlinux.org/index.php/Pe … ge_devices
and lowering a virtual memory to, eg.

vm.dirty_background_bytes = 16777216
vm.dirty_bytes = 16777216

https://wiki.archlinux.org/index.php/Sy … ual_memory

So, those three lines and your two lines all go into a file called /etc/tmpfiles.d/local.conf ?

Last edited by mir91 (2014-10-16 17:30:13)

gedgon

Member

From: Poland

Registered: 2011-01-27

Posts: 95

Re: USB hard drive transfer forces system into a cawl

So, those three lines and your two lines all go into a file called /etc/tmpfiles.d/local.conf ?

/etc/tmpfiles.d/local.conf

w /sys/kernel/mm/transparent_hugepage/enabled - - - - madvise
w /sys/kernel/mm/transparent_hugepage/defrag - - - - madvise
w /sys/kernel/mm/transparent_hugepage/khugepaged/defrag - - - - 0

/etc/sysctl.d/99-sysctl.conf

vm.dirty_background_bytes = 16777216
vm.dirty_bytes = 16777216