You are not logged in.

Pages: 1

#1 2015-01-03 12:51:51

theking2
Banned
From: Romanshorn Switzerland
Registered: 2009-03-04
Posts: 372

System increasingly unstable

My system has run for over 10 years without unaccounted problems. Recently however the system is less responsive and reports timeouts on interrupt:

[Sat Jan  3 13:45:17 2015] perf interrupt took too long (2537 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
[Sat Jan  3 13:45:21 2015] INFO: task txg_sync:490 blocked for more than 120 seconds.
[Sat Jan  3 13:45:21 2015]       Tainted: P           O   3.17.6-1-ARCH #1
[Sat Jan  3 13:45:21 2015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sat Jan  3 13:45:21 2015] txg_sync        D 0000000000000003     0   490      2 0x00000000
[Sat Jan  3 13:45:21 2015]  ffff88007720bb80 0000000000000046 ffff880073675a90 00000000000135c0
[Sat Jan  3 13:45:21 2015]  ffff88007720bfd8 00000000000135c0 ffff88007bca8a10 ffff880073675a90
[Sat Jan  3 13:45:21 2015]  ffff88004db83b10 0000000000000000 0000000000000046 0000000000000001
[Sat Jan  3 13:45:21 2015] Call Trace:
[Sat Jan  3 13:45:21 2015]  [<ffffffff8109b9c2>] ? default_wake_function+0x12/0x20
[Sat Jan  3 13:45:21 2015]  [<ffffffff810af275>] ? __wake_up_common+0x55/0x90
[Sat Jan  3 13:45:21 2015]  [<ffffffff81539649>] schedule+0x29/0x70
[Sat Jan  3 13:45:21 2015]  [<ffffffff81539924>] io_schedule+0x94/0xf0
[Sat Jan  3 13:45:21 2015]  [<ffffffffa02f1c5c>] cv_wait_common+0xac/0x1b0 [spl]
[Sat Jan  3 13:45:21 2015]  [<ffffffff810af960>] ? __wake_up_sync+0x20/0x20
[Sat Jan  3 13:45:21 2015]  [<ffffffffa02f1db8>] __cv_wait_io+0x18/0x20 [spl]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa040556b>] zio_wait+0x11b/0x1f0 [zfs]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa038ed11>] dsl_pool_sync+0xc1/0x470 [zfs]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa03a9970>] spa_sync+0x470/0xb90 [zfs]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa03bad5a>] txg_sync_thread+0x3da/0x680 [zfs]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa03ba980>] ? txg_delay+0x150/0x150 [zfs]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa02e9cea>] thread_generic_wrapper+0x7a/0x90 [spl]
[Sat Jan  3 13:45:21 2015]  [<ffffffffa02e9c70>] ? __thread_exit+0xa0/0xa0 [spl]
[Sat Jan  3 13:45:21 2015]  [<ffffffff8108e2ea>] kthread+0xea/0x100
[Sat Jan  3 13:45:21 2015]  [<ffffffff811faef0>] ? end_buffer_async_write+0x210/0x210
[Sat Jan  3 13:45:21 2015]  [<ffffffff8108e200>] ? kthread_create_on_node+0x1b0/0x1b0
[Sat Jan  3 13:45:21 2015]  [<ffffffff8153dbbc>] ret_from_fork+0x7c/0xb0
[Sat Jan  3 13:45:21 2015]  [<ffffffff8108e200>] ? kthread_create_on_node+0x1b0/0x1b0
[Sat Jan  3 13:49:21 2015] INFO: task txg_sync:490 blocked for more than 120 seconds.
[Sat Jan  3 13:49:21 2015]       Tainted: P           O   3.17.6-1-ARCH #1
[Sat Jan  3 13:49:21 2015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sat Jan  3 13:49:21 2015] txg_sync        D 0000000000000003     0   490      2 0x00000000
[Sat Jan  3 13:49:21 2015]  ffff88007720bb80 0000000000000046 ffff880073675a90 00000000000135c0
[Sat Jan  3 13:49:21 2015]  ffff88007720bfd8 00000000000135c0 ffff88007bca8a10 ffff880073675a90
[Sat Jan  3 13:49:21 2015]  00000000000135c0 ffff88007f200000 0000000000000046 0000000000000001
[Sat Jan  3 13:49:21 2015] Call Trace:
[Sat Jan  3 13:49:21 2015]  [<ffffffff8109b9c2>] ? default_wake_function+0x12/0x20
[Sat Jan  3 13:49:21 2015]  [<ffffffff810af275>] ? __wake_up_common+0x55/0x90
[Sat Jan  3 13:49:21 2015]  [<ffffffff81539649>] schedule+0x29/0x70
[Sat Jan  3 13:49:21 2015]  [<ffffffff81539924>] io_schedule+0x94/0xf0
[Sat Jan  3 13:49:21 2015]  [<ffffffffa02f1c5c>] cv_wait_common+0xac/0x1b0 [spl]
[Sat Jan  3 13:49:21 2015]  [<ffffffff810af960>] ? __wake_up_sync+0x20/0x20
[Sat Jan  3 13:49:21 2015]  [<ffffffffa02f1db8>] __cv_wait_io+0x18/0x20 [spl]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa040556b>] zio_wait+0x11b/0x1f0 [zfs]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa038ed11>] dsl_pool_sync+0xc1/0x470 [zfs]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa03a9970>] spa_sync+0x470/0xb90 [zfs]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa03bad5a>] txg_sync_thread+0x3da/0x680 [zfs]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa03ba980>] ? txg_delay+0x150/0x150 [zfs]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa02e9cea>] thread_generic_wrapper+0x7a/0x90 [spl]
[Sat Jan  3 13:49:21 2015]  [<ffffffffa02e9c70>] ? __thread_exit+0xa0/0xa0 [spl]
[Sat Jan  3 13:49:21 2015]  [<ffffffff8108e2ea>] kthread+0xea/0x100
[Sat Jan  3 13:49:21 2015]  [<ffffffff811faef0>] ? end_buffer_async_write+0x210/0x210
[Sat Jan  3 13:49:21 2015]  [<ffffffff8108e200>] ? kthread_create_on_node+0x1b0/0x1b0
[Sat Jan  3 13:49:21 2015]  [<ffffffff8153dbbc>] ret_from_fork+0x7c/0xb0
[Sat Jan  3 13:49:21 2015]  [<ffffffff8108e200>] ? kthread_create_on_node+0x1b0/0x1b0
[Sat Jan  3 13:49:33 2015] perf interrupt took too long (5004 > 4960), lowering kernel.perf_event_max_sample_rate to 25200

What could be the cause of this degradation? I do see the problem occur with high volume on Samba / ZFS activity. I've included a separate log and cache device.

pool: zdata
 state: ONLINE
  scan: scrub repaired 387K in 11h56m with 0 errors on Sat Jan  3 07:26:16 2015
config:

        NAME                                           STATE     READ WRITE CKSUM
        zdata                                          ONLINE       0     0     0
          raidz1-0                                     ONLINE       0     0     0
            ata-WDC_WD10EADS-00P8B0_WD-WMAVU0350318    ONLINE       0     0     0
            ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW36807048  ONLINE       0     0     0
            ata-WDC_WD10EADS-00P8B0_WD-WMAVU0606317    ONLINE       0     0     0
            ata-WDC_WD10EADS-00P8B0_WD-WCAVU0381916    ONLINE       0     0     0
        logs
          sde5                                         ONLINE       0     0     0
        cache
          sde6                                         ONLINE       0     0     0

errors: No known data errors

archlinux on a Gigabyte C1037UN-EU, 16GiB
a Promise  PDC40718 based ZFS set
root on a Samsung SSD PB22-J
running LogitechMediaServer(-git), Samba, MiniDLNA, TOR

Offline

#2 2015-01-03 12:57:20

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,732
Website

Re: System increasingly unstable

Any reallocated sectors on your disks?

AUR packages • Zsh and other configs

Offline

#3 2015-01-03 13:27:49

theking2
Banned
From: Romanshorn Switzerland
Registered: 2009-03-04
Posts: 372

Re: System increasingly unstable

I don't think so:

[root@janus /]# smartctl -a /dev/sda | grep -i reallocated
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
[root@janus /]# smartctl -a /dev/sdb | grep -i reallocated
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
[root@janus /]# smartctl -a /dev/sdc | grep -i reallocated
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
^[[A[root@janus smartctl -a /dev/sdd | grep -i reallocated
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
[root@janus /]# smartctl -a /dev/sde | grep -i reallocated

Last edited by theking2 (2015-01-04 14:43:36)

archlinux on a Gigabyte C1037UN-EU, 16GiB
a Promise  PDC40718 based ZFS set
root on a Samsung SSD PB22-J
running LogitechMediaServer(-git), Samba, MiniDLNA, TOR

Offline

Pages: 1

Board footer

Powered by FluxBB