Arch Linux

djthread

Member

Registered: 2010-02-14

Posts: 23

kernel bugs in 3.3.2 related to mdadm?

I've been able to fairly consistently replicate this issue by simply booting and making use of my mdadm-controlled raid5 volume. It seems that reads on certain directories hang forever, and I see funny business in my dmesg.

What follows is a few separate instances of the messages, with a reboot between each.

Is this something I should be reporting to bugzilla.kernel.org?

Thanks!

[ 3600.759684] INFO: task jbd2/md0-8:1366 blocked for more than 120 seconds.
[ 3600.759705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3600.759732] jbd2/md0-8      D ffff8801112413c8     0  1366      2 0x00000000
[ 3600.759736]  ffff88011126bce0 0000000000000046 ffff880111241000 ffff88011126bfd8
[ 3600.759740]  ffff88011126bfd8 ffff88011126bfd8 ffffffff8180d020 ffff880111241000
[ 3600.759744]  0000000000013ec0 0000000000000000 ffff88011126bc40 ffffffff810855e8
[ 3600.759747] Call Trace:
[ 3600.759755]  [<ffffffff810855e8>] ? sched_clock_cpu+0xa8/0x120
[ 3600.759759]  [<ffffffff810880a6>] ? update_shares+0xf6/0x120
[ 3600.759764]  [<ffffffff8145e11f>] schedule+0x3f/0x60
[ 3600.759780]  [<ffffffffa008e036>] jbd2_journal_commit_transaction+0x1c6/0x1850 [jbd2]
[ 3600.759785]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[ 3600.759791]  [<ffffffffa0093757>] kjournald2+0xb7/0x240 [jbd2]
[ 3600.759794]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[ 3600.759799]  [<ffffffffa00936a0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 3600.759802]  [<ffffffff81071bf3>] kthread+0x93/0xa0
[ 3600.759805]  [<ffffffff81461264>] kernel_thread_helper+0x4/0x10
[ 3600.759808]  [<ffffffff81071b60>] ? kthread_freezable_should_stop+0x70/0x70
[ 3600.759811]  [<ffffffff81461260>] ? gs_change+0x13/0x13
[ 3720.759668] INFO: task jbd2/md0-8:1366 blocked for more than 120 seconds.
[ 3720.759689] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3720.759715] jbd2/md0-8      D ffff8801112413c8     0  1366      2 0x00000000
[ 3720.759720]  ffff88011126bce0 0000000000000046 ffff880111241000 ffff88011126bfd8
[ 3720.759723]  ffff88011126bfd8 ffff88011126bfd8 ffffffff8180d020 ffff880111241000
[ 3720.759727]  0000000000013ec0 0000000000000000 ffff88011126bc40 ffffffff810855e8
[ 3720.759730] Call Trace:
[ 3720.759738]  [<ffffffff810855e8>] ? sched_clock_cpu+0xa8/0x120
[ 3720.759742]  [<ffffffff810880a6>] ? update_shares+0xf6/0x120
[ 3720.759746]  [<ffffffff8145e11f>] schedule+0x3f/0x60
[ 3720.759761]  [<ffffffffa008e036>] jbd2_journal_commit_transaction+0x1c6/0x1850 [jbd2]
[ 3720.759766]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[ 3720.759771]  [<ffffffffa0093757>] kjournald2+0xb7/0x240 [jbd2]
[ 3720.759775]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[ 3720.759779]  [<ffffffffa00936a0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 3720.759782]  [<ffffffff81071bf3>] kthread+0x93/0xa0
[ 3720.759785]  [<ffffffff81461264>] kernel_thread_helper+0x4/0x10
[ 3720.759789]  [<ffffffff81071b60>] ? kthread_freezable_should_stop+0x70/0x70
[ 3720.759791]  [<ffffffff81461260>] ? gs_change+0x13/0x13


[  240.753346] INFO: task jbd2/md0-8:1386 blocked for more than 120 seconds.
[  240.753412] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.753475] jbd2/md0-8      D ffff88010c643db0     0  1386      2 0x00000000
[  240.753480]  ffff88010c643ce0 0000000000000046 ffff88010c572800 ffff88010c643fd8
[  240.753484]  ffff88010c643fd8 ffff88010c643fd8 ffff88010c484000 ffff88010c572800
[  240.753487]  0000000000013ec0 ffff880117686b60 0000000000000002 0000000000000000
[  240.753490] Call Trace:
[  240.753498]  [<ffffffff810725b0>] ? autoremove_wake_function+0x40/0x40
[  240.753503]  [<ffffffff8119c72e>] ? __wait_on_buffer+0x2e/0x30
[  240.753506]  [<ffffffff8119dafb>] ? __sync_dirty_buffer+0x9b/0xd0
[  240.753510]  [<ffffffff8145e11f>] schedule+0x3f/0x60
[  240.753526]  [<ffffffffa00f5036>] jbd2_journal_commit_transaction+0x1c6/0x1850 [jbd2]
[  240.753530]  [<ffffffff81080ae2>] ? default_wake_function+0x12/0x20
[  240.753533]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[  240.753539]  [<ffffffffa00fa757>] kjournald2+0xb7/0x240 [jbd2]
[  240.753542]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[  240.753547]  [<ffffffffa00fa6a0>] ? commit_timeout+0x10/0x10 [jbd2]
[  240.753569]  [<ffffffff81071bf3>] kthread+0x93/0xa0
[  240.753572]  [<ffffffff81461264>] kernel_thread_helper+0x4/0x10
[  240.753575]  [<ffffffff81071b60>] ? kthread_freezable_should_stop+0x70/0x70
[  240.753578]  [<ffffffff81461260>] ? gs_change+0x13/0x13


[  240.746681] INFO: task jbd2/md0-8:1370 blocked for more than 120 seconds.
[  240.746746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.746808] jbd2/md0-8      D ffff880111d1bbc8     0  1370      2 0x00000000
[  240.746813]  ffff880111da7ce0 0000000000000046 ffff880111d1b800 ffff880111da7fd8
[  240.746817]  ffff880111da7fd8 ffff880111da7fd8 ffffffff8180d020 ffff880111d1b800
[  240.746820]  ffff880111da7c20 ffff88011862c548 0000000000000002 0000000000000000
[  240.746823] Call Trace:
[  240.746831]  [<ffffffff810725b0>] ? autoremove_wake_function+0x40/0x40
[  240.746835]  [<ffffffff8119c72e>] ? __wait_on_buffer+0x2e/0x30
[  240.746839]  [<ffffffff8119dafb>] ? __sync_dirty_buffer+0x9b/0xd0
[  240.746842]  [<ffffffff8145e11f>] schedule+0x3f/0x60
[  240.746857]  [<ffffffffa00f2036>] jbd2_journal_commit_transaction+0x1c6/0x1850 [jbd2]
[  240.746861]  [<ffffffff81080ae2>] ? default_wake_function+0x12/0x20
[  240.746865]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[  240.746870]  [<ffffffffa00f7757>] kjournald2+0xb7/0x240 [jbd2]
[  240.746873]  [<ffffffff81072570>] ? abort_exclusive_wait+0xb0/0xb0
[  240.746878]  [<ffffffffa00f76a0>] ? commit_timeout+0x10/0x10 [jbd2]
[  240.746881]  [<ffffffff81071bf3>] kthread+0x93/0xa0
[  240.746884]  [<ffffffff81461264>] kernel_thread_helper+0x4/0x10
[  240.746887]  [<ffffffff81071b60>] ? kthread_freezable_should_stop+0x70/0x70
[  240.746890]  [<ffffffff81461260>] ? gs_change+0x13/0x13

bluewind

Administrator

From: Austria

Registered: 2008-07-13

Posts: 172

Website

Re: kernel bugs in 3.3.2 related to mdadm?

I'm experiencing something similar on my file server with 2 raid1 arrays combined using lvm.

It seems to happen only once after a reboot (don't count on that though) and the kernel.org bugzilla is likely a better place for this, as it appears to be an upstream issue.

I'm currently running linux 3.3.5-1 and I haven't yet tried the current git nor do I really have the time to start bisecting if it isn't fixed already. If you want to try that yourself take a look at man git-bisect.

[ 9217.177813] INFO: task jbd2/dm-2-8:381 blocked for more than 120 seconds.
[ 9217.178103] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.178372] jbd2/dm-2-8     D ffff880076877bc8     0   381      2 0x00000000
[ 9217.178379]  ffff88007abd9ce0 0000000000000046 ffff880076877800 ffff88007abd9fd8
[ 9217.178386]  ffff88007abd9fd8 ffff88007abd9fd8 ffff88007ba37800 ffff880076877800
[ 9217.178391]  ffff88007f0134f8 00000000000078c2 ffff88007abd9c60 ffffffff81087c0c
[ 9217.178397] Call Trace:
[ 9217.178409]  [<ffffffff81087c0c>] ? update_curr+0xec/0x1b0
[ 9217.178414]  [<ffffffff810882e2>] ? dequeue_entity+0xb2/0x220
[ 9217.178420]  [<ffffffff8108610f>] ? __dequeue_entity+0x2f/0x50
[ 9217.178426]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.178445]  [<ffffffffa0129036>] jbd2_journal_commit_transaction+0x1c6/0x1850 [jbd2]
[ 9217.178454]  [<ffffffff81072540>] ? abort_exclusive_wait+0xb0/0xb0
[ 9217.178462]  [<ffffffffa012e757>] kjournald2+0xb7/0x240 [jbd2]
[ 9217.178467]  [<ffffffff81072540>] ? abort_exclusive_wait+0xb0/0xb0
[ 9217.178474]  [<ffffffffa012e6a0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 9217.178480]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.178485]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.178490]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.178495]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.178501] INFO: task nfsd:729 blocked for more than 120 seconds.
[ 9217.178769] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.179042] nfsd            D ffff88007593d3c8     0   729      2 0x00000000
[ 9217.179047]  ffff88007a23f8b0 0000000000000046 ffff88007593d000 ffff88007a23ffd8
[ 9217.179052]  ffff88007a23ffd8 ffff88007a23ffd8 ffffffff8180d020 ffff88007593d000
[ 9217.179058]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff880079577500
[ 9217.179063] Call Trace:
[ 9217.179069]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.179075]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.179079]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.179084]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.179089]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.179094]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.179100]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.179112]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.179118]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.179123]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.179132]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.179139]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.179143]  [<ffffffff8118438e>] ? d_obtain_alias+0x5e/0x250
[ 9217.179149]  [<ffffffff81156ac9>] ? __kmalloc+0x39/0x1a0
[ 9217.179158]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.179164]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.179169]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.179174]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.179179]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.179188]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.179193]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.179201]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.179206]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.179214]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.179222]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.179232]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.179239]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.179250]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.179257]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.179264]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.179269]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.179274]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.179279]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.179283]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.179287] INFO: task nfsd:731 blocked for more than 120 seconds.
[ 9217.179567] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.179849] nfsd            D ffff88007593abc8     0   731      2 0x00000000
[ 9217.179854]  ffff88007ae178b0 0000000000000046 ffff88007593a800 ffff88007ae17fd8
[ 9217.179860]  ffff88007ae17fd8 ffff88007ae17fd8 ffff88007ba37800 ffff88007593a800
[ 9217.179865]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff880074f7ae00
[ 9217.179871] Call Trace:
[ 9217.179875]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.179880]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.179884]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.179889]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.179893]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.179898]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.179903]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.179912]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.179918]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.179922]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.179931]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.179936]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.179941]  [<ffffffff8118438e>] ? d_obtain_alias+0x5e/0x250
[ 9217.179946]  [<ffffffff81156ac9>] ? __kmalloc+0x39/0x1a0
[ 9217.179953]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.179958]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.179963]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.179968]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.179973]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.179982]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.179987]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.179995]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.180000]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.180008]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.180016]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.180025]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.180032]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.180043]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.180049]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.180054]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.180059]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.180064]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.180069]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.180073]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.180077] INFO: task nfsd:732 blocked for more than 120 seconds.
[ 9217.180362] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.180653] nfsd            D ffff88007593b3c8     0   732      2 0x00000000
[ 9217.180658]  ffff880074def8b0 0000000000000046 ffff88007593b000 ffff880074deffd8
[ 9217.180663]  ffff880074deffd8 ffff880074deffd8 ffffffff8180d020 ffff88007593b000
[ 9217.180669]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff880074f7a300
[ 9217.180674] Call Trace:
[ 9217.180678]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.180683]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.180688]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.180692]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.180697]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.180701]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.180707]  [<ffffffff8135b41b>] ? skb_copy_datagram_iovec+0x5b/0x2a0
[ 9217.180711]  [<ffffffff81357d97>] ? __kfree_skb+0x47/0xa0
[ 9217.180720]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.180725]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.180730]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.180738]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.180744]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.180749]  [<ffffffff81156bd7>] ? __kmalloc+0x147/0x1a0
[ 9217.180756]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.180761]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.180766]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.180771]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.180776]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.180785]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.180790]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.180798]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.180803]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.180811]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.180819]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.180828]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.180835]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.180845]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.180852]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.180857]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.180862]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.180866]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.180872]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.180876]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.180879] INFO: task nfsd:733 blocked for more than 120 seconds.
[ 9217.181255] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.181576] nfsd            D ffff88007593c3c8     0   733      2 0x00000000
[ 9217.181587]  ffff880074e1d8b0 0000000000000046 ffff88007593c000 ffff880074e1dfd8
[ 9217.181600]  ffff880074e1dfd8 ffff880074e1dfd8 ffff88007ba37800 ffff88007593c000
[ 9217.181613]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff880074f7b800
[ 9217.181628] Call Trace:
[ 9217.181635]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.181645]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.181653]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.181662]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.181671]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.181679]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.181688]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.181703]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.181712]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.181720]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.181733]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.181743]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.181752]  [<ffffffff8118438e>] ? d_obtain_alias+0x5e/0x250
[ 9217.181761]  [<ffffffff81156ac9>] ? __kmalloc+0x39/0x1a0
[ 9217.181772]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.181781]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.181790]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.181799]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.181808]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.181821]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.181831]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.181841]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.181851]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.181863]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.181874]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.181887]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.181899]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.181914]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.181925]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.181934]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.181942]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.181950]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.181960]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.181968]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.181974] INFO: task nfsd:734 blocked for more than 120 seconds.
[ 9217.182284] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.182591] nfsd            D ffff880075938bc8     0   734      2 0x00000000
[ 9217.182596]  ffff880074e658b0 0000000000000046 ffff880075938800 ffff880074e65fd8
[ 9217.182602]  ffff880074e65fd8 ffff880074e65fd8 ffffffff8180d020 ffff880075938800
[ 9217.182607]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff88007a44a800
[ 9217.182612] Call Trace:
[ 9217.182617]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.182622]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.182626]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.182631]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.182635]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.182640]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.182645]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.182654]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.182659]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.182664]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.182672]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.182678]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.182682]  [<ffffffff8118438e>] ? d_obtain_alias+0x5e/0x250
[ 9217.182687]  [<ffffffff81156ac9>] ? __kmalloc+0x39/0x1a0
[ 9217.182694]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.182699]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.182704]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.182709]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.182714]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.182723]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.182729]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.182736]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.182741]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.182749]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.182757]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.182766]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.182774]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.182784]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.182791]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.182795]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.182800]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.182805]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.182810]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.182815]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.182818] INFO: task nfsd:735 blocked for more than 120 seconds.
[ 9217.183125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.183454] nfsd            D ffff88007593fbc8     0   735      2 0x00000000
[ 9217.183463]  ffff880074ead8b0 0000000000000046 ffff88007593f800 ffff880074eadfd8
[ 9217.183469]  ffff880074eadfd8 ffff880074eadfd8 ffffffff8180d020 ffff88007593f800
[ 9217.183474]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff88007a44b900
[ 9217.183480] Call Trace:
[ 9217.183484]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.183489]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.183494]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.183498]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.183503]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.183507]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.183512]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.183522]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.183527]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.183531]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.183540]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.183545]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.183550]  [<ffffffff8118438e>] ? d_obtain_alias+0x5e/0x250
[ 9217.183555]  [<ffffffff81156ac9>] ? __kmalloc+0x39/0x1a0
[ 9217.183562]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.183567]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.183572]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.183578]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.183582]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.183591]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.183597]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.183604]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.183609]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.183617]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.183625]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.183634]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.183642]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.183652]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.183659]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.183663]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.183668]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.183673]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.183678]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.183683]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9217.183686] INFO: task nfsd:736 blocked for more than 120 seconds.
[ 9217.183996] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9217.184325] nfsd            D ffff88007593e3c8     0   736      2 0x00000000
[ 9217.184331]  ffff880074eed8b0 0000000000000046 ffff88007593e000 ffff880074eedfd8
[ 9217.184336]  ffff880074eedfd8 ffff880074eedfd8 ffff88007ba37800 ffff88007593e000
[ 9217.184342]  ffff8800798a1c00 ffff880079612000 ffff88007adcac9c ffff880074f7b600
[ 9217.184347] Call Trace:
[ 9217.184352]  [<ffffffff81381cda>] ? sch_direct_xmit+0x6a/0x1d0
[ 9217.184356]  [<ffffffff81366367>] ? dev_queue_xmit+0x217/0x6c0
[ 9217.184361]  [<ffffffff8145e0df>] schedule+0x3f/0x60
[ 9217.184365]  [<ffffffff8145ca30>] __mutex_lock_slowpath+0x160/0x370
[ 9217.184370]  [<ffffffff8145cc56>] mutex_lock+0x16/0x30
[ 9217.184375]  [<ffffffff8110b9d5>] generic_file_aio_write+0x55/0xe0
[ 9217.184380]  [<ffffffff813b3317>] ? tcp_transmit_skb+0x3a7/0x890
[ 9217.184389]  [<ffffffffa0147f3f>] ext4_file_write+0xaf/0x260 [ext4]
[ 9217.184395]  [<ffffffff81186bc9>] ? iget_locked+0x89/0x190
[ 9217.184399]  [<ffffffff811815db>] ? __d_find_any_alias+0x4b/0x70
[ 9217.184408]  [<ffffffffa0147e90>] ? ext4_file_mmap+0x70/0x70 [ext4]
[ 9217.184414]  [<ffffffff8116c9b2>] do_sync_readv_writev+0xd2/0x110
[ 9217.184419]  [<ffffffff81156bd7>] ? __kmalloc+0x147/0x1a0
[ 9217.184427]  [<ffffffffa06783a0>] ? _fh_update.isra.8.part.9+0x60/0x60 [nfsd]
[ 9217.184432]  [<ffffffff811f518c>] ? security_file_permission+0x2c/0xb0
[ 9217.184437]  [<ffffffff8116c0d1>] ? rw_verify_area+0x61/0xf0
[ 9217.184442]  [<ffffffff8116cc9b>] do_readv_writev+0xdb/0x1f0
[ 9217.184447]  [<ffffffff811c7c53>] ? dquot_initialize+0x13/0x20
[ 9217.184456]  [<ffffffffa0147cd2>] ? ext4_file_open+0x72/0x1c0 [ext4]
[ 9217.184461]  [<ffffffff8116cde5>] vfs_writev+0x35/0x60
[ 9217.184469]  [<ffffffffa067986b>] nfsd_vfs_write.isra.11+0xeb/0x3e0 [nfsd]
[ 9217.184474]  [<ffffffff8116a8ff>] ? dentry_open+0x4f/0x90
[ 9217.184482]  [<ffffffffa067a843>] ? nfsd_open+0xa3/0x180 [nfsd]
[ 9217.184490]  [<ffffffffa067bf78>] nfsd_write+0xf8/0x110 [nfsd]
[ 9217.184499]  [<ffffffffa0683a8b>] nfsd3_proc_write+0xbb/0x150 [nfsd]
[ 9217.184507]  [<ffffffffa0675a3b>] nfsd_dispatch+0xeb/0x230 [nfsd]
[ 9217.184517]  [<ffffffffa0590bab>] svc_process+0x4ab/0x830 [sunrpc]
[ 9217.184524]  [<ffffffffa06750b5>] nfsd+0xb5/0x150 [nfsd]
[ 9217.184529]  [<ffffffffa0675000>] ? 0xffffffffa0674fff
[ 9217.184534]  [<ffffffff81071bc3>] kthread+0x93/0xa0
[ 9217.184538]  [<ffffffff81461224>] kernel_thread_helper+0x4/0x10
[ 9217.184544]  [<ffffffff81071b30>] ? kthread_freezable_should_stop+0x70/0x70
[ 9217.184548]  [<ffffffff81461220>] ? gs_change+0x13/0x13
[ 9263.826103] rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
[ 9263.826960] nfsd: peername failed (err 107)!