system hangs when copying data to usb device

steve_v · 2011-12-25 00:26:39

Hrmm, I for one like a usable keyboard @ bootup, besides this doesn't appear to improve matters here.

Just to clarify, can someone run a 'dd if=/dev/zero of=/somewhere/on/usb/drive -bs=1M -count=500'
For me this does NOT cause a problem.
Nor does copying said file back & forth from the usb stick to a sata device.

However, accessing that usb drive with dolphin while the write is in progress hangs up the whole desktop...
Accessing the drive from a console (ls, cat, mc etc.) is fine.

Anyone care to confirm, or am i dealing with yet annother different issue...

Steve.

lilsirecho · 2011-12-25 01:24:31

FWIW:
Perhaps this occurs with USB flash devices which are connected via USB 1.1 reader/writers.

Also, some USB devices reader devices are usb2.0 and are compatible with true IDE flash devices and may not respond well to devices which are not true IDE and thereby cause the supposed i/o problem.EDIT: I comment on CF devices with this parameter. I do not know of a similar designator for SD devices. Perhaps trying CF devices could provide better results?

In my system, true IDE devices can be booted but those which are not true IDE will fail to boot.

Perhaps this isn't relevant for your varied performance problems.

Last edited by lilsirecho (2011-12-25 02:00:03)

stqn · 2011-12-27 11:30:53

For me disabling transparent huge pages helped (no more freeze, faster copy to usb flash drive.)
https://bbs.archlinux.org/viewtopic.php … 9#p1032819

finidine · 2011-12-27 19:11:31

Now trying with this:

echo never >/sys/kernel/mm/transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

And yes, seems to help a lot. Multiple and very long USB transfers, system keeps responsive.

Draje · 2011-12-27 22:53:51

Disabling THP also seems to solve my problem. I no longer see other applications entering D-state when running a file copy.

firecat53 · 2011-12-27 23:28:58

So, if we disable this hugepage option, what are the possible ramifications for other areas of the system?

Scott

Draje · 2011-12-28 10:10:49

firecat53 wrote:

So, if we disable this hugepage option, what are the possible ramifications for other areas of the system?
Scott

Actually, hugepages do not need to be disabled, but only the defragmentation of them. Adding the following to the rc.local will set it on boot.

echo never > /sys/kernel/mm/transparent_hugepage/defrag

Apart from fixing the file copy issue, I'm not sure how else the system is affected. I've noticed only benefits from disabling the defragmentation.

unikum · 2011-12-28 11:45:54

Draje, can I take a same effect with sysctl.conf?

finidine · 2011-12-28 13:44:47

In fact,

echo madvise > /sys/kernel/mm/transparent_hugepage/defrag

may be enough (to not disable hugepage defrag completely).

kokoko3k · 2011-12-29 07:24:23

Draje wrote:

firecat53 wrote:
So, if we disable this hugepage option, what are the possible ramifications for other areas of the system?
Scott
Actually, hugepages do not need to be disabled, but only the defragmentation of them. Adding the following to the rc.local will set it on boot.
echo never > /sys/kernel/mm/transparent_hugepage/defrag
Apart from fixing the file copy issue, I'm not sure how else the system is affected. I've noticed only benefits from disabling the defragmentation.

Well, yesterday my netbook refused to enter hibernation state due to segfaults, it worked since months. (and hugepaged defrag appeared in the backtrace)
Re-enabling the defragmentation solved it, now i enable it just before hibernation and disable at wake up.

EDIT
somebody tested madvise instead of never?

Last edited by kokoko3k (2011-12-29 07:26:17)

finidine · 2011-12-29 11:39:32

kokoko3k wrote:

EDIT
somebody tested madvise instead of never?

I'm testing it. Good responsiveness, no crashes / any other issues.

Just for reference, the Ubuntu Oneiric (current) kernel has this default configuration:

transparent_hugepage: madvise
hugepage_defrag: always
khugepaged: enabled (1)

Currently I'm testing this:
transparent_hugepage: always
hugepage_defrag: madvise
khugepaged: enabled (1)

Which, to me, is less restrictive.

stqn · 2011-12-29 19:42:00

unikum wrote:

Draje, can I take a same effect with sysctl.conf?

sysctl's man page states that it can only affect /proc/sys settings, and indeed when I tried to change /sys/kernel/mm/transparent_hugepage/enabled with sysctl.conf, it didn't work.

finidine · 2012-01-02 23:22:50

This is a good reading on the subject:

http://lwn.net/Articles/467328/

Also, note that the default value of /sys/kernel/mm/transparent_hugepage/enabled can be defined at kernel compile time.

pyther · 2012-01-15 16:12:52

Thank you so much!!! Using madvise and things are working so much better!

kokoko3k · 2012-01-15 17:35:18

Any hope this will be the default in arch stock kernel?

karol · 2012-01-15 17:48:41

I think you can open a feature request.

kokoko3k · 2012-01-16 07:37:47

https://bugs.archlinux.org/task/27957

toofishes · 2012-01-31 01:33:59

We don't patch or change the defaults of the kernel except in very rare cases, and this is not one of them. This bug needs to get resolved upstream, so make sure it is reported on the right channels there.

pyther · 2012-01-31 01:44:51

There have been some significant changes made to the memory compaction code to avoid the lengthy stalls experienced by some users when writing data to slow devices (USB keys, for example). This problem was described in this article, but the solution has evolved considerably. By making a number of changes to how compaction works, the memory management hackers (and Mel Gorman in particular) were able to avoid disabling synchronous compaction, which had the unfortunate effect of reducing huge page usage. See this commit for a lot of information on how this problem was addressed.

https://lwn.net/Articles/475405/

So the problem might be much improved in 3.3

bwat47 · 2012-02-09 23:40:53

I was having this same problem, either cropped up after a recent kernel upgrade or the fact that I switched to kde. copying a ~1.5 gig file to a flash drive caused kde to totally freeze near the end of the transfer for a good 30 seconds. using madvise in rc.local as suggested here fixed it.

XD_dued · 2012-02-13 05:02:08

madvise helped a a lot, made the occurances much less and of less duration.

Unfortunately, it didn't completely eliminate the problem. I will try the never option I guess.

I'm using x86_64 with kde.

Last edited by XD_dued (2012-02-13 05:03:57)

XD_dued · 2012-02-14 23:56:16

For me to almost completely fix the issue, I had to disable legacy usb support in bios and use madvise in the rc.local.

Unfortunately, the problem still happened, but only when I forced it to happen (opened libreoffice and firefox at the same time). And usb transfer is still very slow.

How big would be the disadvantage to try to disable the hugepage defragmentation?

How about to completely disable hugepages?

sxe · 2012-02-15 10:01:55

XD_dued wrote:

How big would be the disadvantage to try to disable the hugepage defragmentation?

I figured no disadvantage by doing so. I have disabled it for some month now, works great and no lag at all anymore.

achilleas.k · 2012-02-16 23:06:21

Disabling hugepage defrag helps. I wont say it fixes the situation because there's still some intermittent freezes of 2-3 seconds. They are especially noticeable in Firefox. But overall it's at least usable now.
Will try with madvise as well.

finidine · 2012-02-17 12:22:56

We have to distinguish between disabling transparent hugepages and disabling hugepage defragmentation.

If you disable transparent hugepages completely, that will have some side effects. For example, hibernation will not work.

"madvise" for transparent hugepages and "enable" (default) for hugepage defragmentation seems to be the best workaround settings until this issue is fixed upstream.

Arch Linux

#76 2011-12-25 00:26:39

Re: system hangs when copying data to usb device

#77 2011-12-25 01:24:31

Re: system hangs when copying data to usb device

#78 2011-12-27 11:30:53

Re: system hangs when copying data to usb device

#79 2011-12-27 19:11:31

Re: system hangs when copying data to usb device

#80 2011-12-27 22:53:51

Re: system hangs when copying data to usb device

#81 2011-12-27 23:28:58

Re: system hangs when copying data to usb device

#82 2011-12-28 10:10:49

Re: system hangs when copying data to usb device

#83 2011-12-28 11:45:54

Re: system hangs when copying data to usb device

#84 2011-12-28 13:44:47

Re: system hangs when copying data to usb device

#85 2011-12-29 07:24:23

Re: system hangs when copying data to usb device

#86 2011-12-29 11:39:32

Re: system hangs when copying data to usb device

#87 2011-12-29 19:42:00

Re: system hangs when copying data to usb device

#88 2012-01-02 23:22:50

Re: system hangs when copying data to usb device

#89 2012-01-15 16:12:52

Re: system hangs when copying data to usb device

#90 2012-01-15 17:35:18

Re: system hangs when copying data to usb device

#91 2012-01-15 17:48:41

Re: system hangs when copying data to usb device

#92 2012-01-16 07:37:47

Re: system hangs when copying data to usb device

#93 2012-01-31 01:33:59

Re: system hangs when copying data to usb device

#94 2012-01-31 01:44:51

Re: system hangs when copying data to usb device

#95 2012-02-09 23:40:53

Re: system hangs when copying data to usb device

#96 2012-02-13 05:02:08

Re: system hangs when copying data to usb device

#97 2012-02-14 23:56:16

Re: system hangs when copying data to usb device

#98 2012-02-15 10:01:55

Re: system hangs when copying data to usb device

#99 2012-02-16 23:06:21

Re: system hangs when copying data to usb device

#100 2012-02-17 12:22:56

Re: system hangs when copying data to usb device

Board footer