System lock-ups or software resets, with increased ethernet activity

wantilles · 2011-03-24 15:04:01

I have been having full system lock-ups (hang-ups) -> ie -> no caps/num lock functioning, no keyboard functioning, no mouse movements, or software resets.

This has been happening during the last week or so, almost every time when there is increased network (ethernet) activity.

Checking the kernel logs of the last week, I found a large amount of groups of entries like this:

Mar 20 19:13:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 19:13:25 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 22:45:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 22:45:17 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 22:45:17 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 22:45:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 22:45:20 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:12 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:13 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:15 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:19 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: NOHZ: local_softirq_pending 08
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:21 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:24 machine kernel: net_ratelimit: 20 callbacks suppressed
Mar 20 23:15:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:26 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:26 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:26 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:26 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:30 machine kernel: net_ratelimit: 16 callbacks suppressed
Mar 20 23:15:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 20 23:15:31 machine kernel: r8169 0000:04:00.0: eth0: link up

Now, at this point there was either a system hang-up, or a software reset because the exact next entries where:

Mar 20 23:16:26 machine kernel: Initializing cgroup subsys cpuset
Mar 20 23:16:26 machine kernel: Initializing cgroup subsys cpu
Mar 20 23:16:26 machine kernel: Linux version 2.6.37-ARCH (tobias@T-POWA-LX) (gcc version 4.5.2 20110127 (prerelease) (GCC) ) #1 SMP PREEMPT Tue Mar 15 09:21:17 CET 2011
Mar 20 23:16:26 machine kernel: Command line: root=/dev/sda2 ro 5 swncq=1 nomodset
Mar 20 23:16:26 machine kernel: BIOS-provided physical RAM map:

Which clearly indicate a reboot (either a self-inflicted software one - or a hardware reset by me, with the reset button).

Ethernet chipset is this one:

Mar 20 19:12:08 machine kernel: r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
Mar 20 19:12:08 machine kernel: r8169 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
Mar 20 19:12:08 machine kernel: r8169 0000:04:00.0: setting latency timer to 64
Mar 20 19:12:08 machine kernel: r8169 0000:04:00.0: irq 46 for MSI/MSI-X
Mar 20 19:12:08 machine kernel: r8169 0000:04:00.0: eth0: RTL8168d/8111d at 0xffffc90004126000, 00:24:1d:c9:e9:74, XID 083000c0 IRQ 46

This issue is critical for me, as I do a lot of work over the LAN here.

Should I file a bug report in the bugzilla, for the kernel?

PS: I checked all kernel logs before a week or so ago, and there is not a single entry like the following:

Mar 20 19:13:25 machine kernel: NOHZ: local_softirq_pending 08

Mar 20 23:15:24 machine kernel: net_ratelimit: XX callbacks suppressed

Last edited by wantilles (2011-03-28 02:01:53)

wantilles · 2011-03-24 15:39:47

To further assist you:

- the first entry of such errors in the logs happened on:

Mar 16 18:38:24

And just before that, pacman log indicates that a kernel upgrade happened:

[2011-03-16 17:29] upgraded kernel26 (2.6.37.3-1 -> 2.6.37.4-1)
[2011-03-16 17:29] upgraded kernel26-headers (2.6.37.3-1 -> 2.6.37.4-1)

Last edited by wantilles (2011-03-28 02:02:18)

wantilles · 2011-03-24 15:58:37

I have rolled-back kernel26 (and headers) to 2.6.37.3-1 and put them in Ignorepkg in pacman.conf.

I will see how it goes, and if I do not have any further issues after a day or two, I will file a kernel bug report at the bugzilla.

darkrex · 2011-03-25 03:46:54

I have exactly the same issue as Wantilles mentioned.
Even the log entries,are identical.

I ll try rolling back to previous kernel version.

wantilles · 2011-03-25 03:53:40

After kernel downgrade/rollback, there have been 12 continuous hours of the usual heavy ethernet usage, with no further incident at all, and completely normal-flawless behavior.

So logic dictates it must have to do with the 2.6.37.4 kernel.

I will open a relevant bug report in the bugzilla, tomorrow.

wantilles · 2011-03-25 16:03:08

Now over 24 hours of downgraded/rollbacked kernel to 2.6.37.3, and with no incidents, and flawless behavior.

wantilles · 2011-03-25 16:03:39

Bug report has been filed:

https://bugs.archlinux.org/task/23429

wantilles · 2011-03-26 16:34:17

Problem remains in 2.6.37.5.

After upgrading:

[2011-03-26 16:10] upgraded kernel26 (2.6.37.3-1 -> 2.6.37.5-1)
[2011-03-26 16:10] upgraded kernel26-headers (2.6.37.3-1 -> 2.6.37.5-1)

Here was the first lockup:

Mar 26 18:16:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:16:44 machine kernel: NOHZ: local_softirq_pending 08

Mar 26 18:19:04 machine kernel: Initializing cgroup subsys cpuset
Mar 26 18:19:04 machine kernel: Initializing cgroup subsys cpu
Mar 26 18:19:04 machine kernel: Linux version 2.6.37-ARCH (tobias@T-POWA-LX) (gcc version 4.5.2 20110127 (prerelease) (GCC) ) #1 SMP PREEMPT Tue Mar 8 08:34:35 CET 2011
Mar 26 18:19:04 machine kernel: Command line: root=/dev/sda2 ro 5 swncq=1 nomodset
Mar 26 18:19:04 machine kernel: BIOS-provided physical RAM map:

And here are the exact same incidents for the 2.5 hours that I was running on the 2.6.37.5 kernel:

Mar 26 16:26:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 16:26:11 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 16:26:11 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 16:26:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 16:26:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 16:26:12 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 16:26:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 16:26:13 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 16:26:13 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 16:26:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 16:26:13 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 17:00:23 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:30 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 17:00:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:30 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 17:00:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:31 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:31 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 17:00:33 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:33 machine kernel: NOHZ: local_softirq_pending 08
Mar 26 17:00:55 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:00:57 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:04:03 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:04:05 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:05:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:01 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:01 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:29 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:29 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:30 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:31 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:11:31 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:19:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:19:27 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:19:27 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:33 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:21:49 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:41:37 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:04 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:05 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:42:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:55 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:56 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:56 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:56 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:57 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:43:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:00 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:07 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:07 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:07 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:09 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:44:09 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:52:29 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 17:59:55 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:00:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:00:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:02:07 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:02:09 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:43 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:43 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:43 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: net_ratelimit: 10 callbacks suppressed
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:48 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:48 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:48 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:48 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:52 machine kernel: net_ratelimit: 23 callbacks suppressed
Mar 26 18:10:52 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:52 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:52 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:52 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:52 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:53 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:53 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:53 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:53 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:53 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:58 machine kernel: net_ratelimit: 22 callbacks suppressed
Mar 26 18:10:58 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:58 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:58 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:10:59 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:13 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:14 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:14 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:15 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:18 machine kernel: net_ratelimit: 21 callbacks suppressed
Mar 26 18:11:18 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:18 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:20 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:21 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:21 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:21 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:21 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:24 machine kernel: net_ratelimit: 12 callbacks suppressed
Mar 26 18:11:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:25 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:26 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:38 machine kernel: net_ratelimit: 4 callbacks suppressed
Mar 26 18:11:38 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:40 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:40 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:40 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:41 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:44 machine kernel: net_ratelimit: 9 callbacks suppressed
Mar 26 18:11:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:46 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:47 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:49 machine kernel: net_ratelimit: 11 callbacks suppressed
Mar 26 18:11:49 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:49 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:49 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:49 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:50 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:11:54 machine kernel: net_ratelimit: 21 callbacks suppressed
Mar 26 18:11:54 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:06 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:07 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:09 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:09 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: net_ratelimit: 1 callbacks suppressed
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:11 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:12 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:17 machine kernel: net_ratelimit: 18 callbacks suppressed
Mar 26 18:12:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:17 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:18 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:18 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:18 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:19 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:22 machine kernel: net_ratelimit: 13 callbacks suppressed
Mar 26 18:12:22 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:22 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:22 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:23 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:23 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:23 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:23 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:24 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:38 machine kernel: net_ratelimit: 10 callbacks suppressed
Mar 26 18:12:38 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:39 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:40 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:40 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:44 machine kernel: net_ratelimit: 14 callbacks suppressed
Mar 26 18:12:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:44 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:12:45 machine kernel: r8169 0000:04:00.0: eth0: link up
Mar 26 18:14:25 machine kernel: Initializing cgroup subsys cpuset
Mar 26 18:14:25 machine kernel: Initializing cgroup subsys cpu
Mar 26 18:14:25 machine kernel: Linux version 2.6.37-ARCH (tobias@T-POWA-LX) (gcc version 4.5.2 20110127 (prerelease) (GCC) ) #1 SMP PREEMPT Fri Mar 25 15:10:00 CET 2011
Mar 26 18:14:25 machine kernel: Command line: root=/dev/sda2 ro 3 swncq=1 nomodset
Mar 26 18:14:25 machine kernel: BIOS-provided physical RAM map:

Again you can see the lockup.

Last edited by wantilles (2011-03-28 02:03:16)

wantilles · 2011-03-26 17:06:02

The same issue, on the linux kernel mailing list:

https://lkml.org/lkml/2011/2/22/316

lydgate · 2011-03-28 18:56:09

I've got the same thing in my logs with r8169 driver, thanks so much for posting this. I was suspecting either nvidia driver or bad RAM, but like you, it started happening shortly after upgrading from 2.6.37.3-1 to 2.6.37.4-1. It was very inconsistent about how often it froze. I don't think I noticed it until Friday morning, when it froze repeatedly. I turned the machine off, expecting a weekend of hardware problems, but actually it was fine over the weekend until last night (when it froze 3-4 times in the final 20 minutes of Peter Bogdanovic's Targets, brilliant timing). This morning and this afternoon it froze a lot too, to the point where I couldn't log in and downgrade the kernel, it would freeze too soon. Admittedly I was SSHing so maybe this exacerbated the problem.

I've just unplugged the network cable, rebooted, and downgraded to 2.6.37.3-1. Will post in 24 hours or so to let you know how I get on, but it's already looking better than it was.

skysurfer · 2011-03-29 23:09:57

Same problem here too (with r8169)

dunz0r · 2011-03-30 00:10:12

Same problem here too, with r8169. I'm going to try to upgrade to 2.6.38 from testing instead of downgrading actually. Will report back to you.

lydgate · 2011-03-30 07:22:57

Just wanted to report that I've been running with 2.6.37.3-1 since a few nights ago and had no issues. Previously I was getting a freeze at unpredictable intervals, but commercial flagging in MythTV seemed to trigger it (the recordings are stored over NFS so I think this basically just means network activitiy triggers it). Now I've recorded several new things and commercial flagging has been fine, so I think it's an issue with 2.6.37.4.

dunz0r · 2011-03-30 09:10:39

My machine has been running stable with 2.6.38 since last night now. We'll see if that is indeed the truth later today, when I can generate some traffic

wantilles · 2011-03-30 16:05:59

dunz0r wrote:

My machine has been running stable with 2.6.38 since last night now. We'll see if that is indeed the truth later today, when I can generate some traffic

Do not rely merely on observation.

Check the kernel logs.

If everything is indeed OK, there should be no entries present, of the following type:

NOHZ: local_softirq_pending 08

net_ratelimit: XX callbacks suppressed

Then, only then, it is really OK.

dunz0r · 2011-04-02 12:40:59

No occurrences of NOHZ: local_softirq_pending 08 so far.

lydgate · 2011-04-09 10:46:59

Anyone tested this with 2.6.37.5?

wantilles · 2011-04-11 12:55:15

lydgate wrote:

Anyone tested this with 2.6.37.5?

Problem remains.

Anyone tested it with 2.6.38.2?

wantilles · 2011-04-11 17:59:21

Problem remains with kernel 2.6.38.2.

fphillips · 2011-04-11 22:50:19

@wantilles

You should reply to the kernel bug report that the problem started between 2.6.37.3 - 2.6.37.4 and any other useful things you can add (this would also generate some traffic so it doesn't get ignored).

https://bugzilla.kernel.org/show_bug.cgi?id=32962

lydgate · 2011-04-28 20:22:56

@fphillips Thanks for the suggestion, I've posted there.

If anyone has tested with anything newer than 2.6.38.2 please let me know. If not I may try git bisect to see if I can figure out what's gone wrong.

lydgate · 2011-04-29 00:03:25

Here is what changed in r8169.c between 2.6.37.3 and 2.6.37.4:
http://dpaste.com/536991/

I am not a programmer but I wonder if upstream commit ba04c7c93bbcb48ce880cf75b6e9dffcd79d4c7b could be the culprit? It claims to fix something that sounds similar to what we're having, but it was also a change introduced at the right time to explain the issue (between Arch 2.6.37.3 and 2.6.37.4):
http://www.kernel.org/pub/linux/kernel/ … -2.6.32.32

Maybe I'll try reverting that in a 2.6.38 kernel and see if I still get the problem.

fphillips · 2011-04-29 03:45:30

That commit went into 2.6.37.3, so it was in the good kernel: http://www.kernel.org/pub/linux/kernel/ … g-2.6.37.3

git-bisect is a good idea. There aren't many commits, and it won't make you go through all of them.

lydgate · 2011-04-29 11:15:29

I'm slightly confused about how the Arch versions line up to the kernel versions. At first I thought that Arch kernel 2.6.37.4 would be the same as Linux kernel 2.6.37 rc4 but I'm not sure that's right? So I just checked out Arch's packages.git and looked at the PKGBUILD for kernel26-2.6.37.4. I checked out Arch packages.git commit d5dd0ba79a(...). In the PKGBUILD it seemed to download Linux 2.6.37 and then patch it using patches from ftp.archlinux.org. I thought I'd found the right patch but I must have got a bit confused.

Anyway, I think maybe git bisect on the Arch Linux tree is a better idea than on the Linux kernel itself which is what I initially tried. Let me know if you have any advice for doing this, the Arch Linux commit messages are a bit confusing to me.

lydgate · 2011-04-29 11:56:07

Ah, ok, I was being stupid. The Arch Linux patches are cumulative, so as fphillips says, that change was introduced earlier.

The changes that were introduced between Arch Linux's 2.6.37.3 and 2.6.37.4 kernels seem to correspond to Linux kernel commits:

0d672e9f8ac320c6d1ea9103db6df7f99ea20361
f60ac8e7ab7cbb413a0131d5665b053f9f386526
1519e57fe81c14bb8fa4855579f19264d1ef63b4
b5ba6d12bdac21bc0620a5089e0f24e362645efd

Does that make more sense?

Last edited by lydgate (2011-04-29 11:56:31)

Arch Linux

#1 2011-03-24 15:04:01

System lock-ups or software resets, with increased ethernet activity

#2 2011-03-24 15:39:47

Re: System lock-ups or software resets, with increased ethernet activity

#3 2011-03-24 15:58:37

Re: System lock-ups or software resets, with increased ethernet activity

#4 2011-03-25 03:46:54

Re: System lock-ups or software resets, with increased ethernet activity

#5 2011-03-25 03:53:40

Re: System lock-ups or software resets, with increased ethernet activity

#6 2011-03-25 16:03:08

Re: System lock-ups or software resets, with increased ethernet activity

#7 2011-03-25 16:03:39

Re: System lock-ups or software resets, with increased ethernet activity

#8 2011-03-26 16:34:17

Re: System lock-ups or software resets, with increased ethernet activity

#9 2011-03-26 17:06:02

Re: System lock-ups or software resets, with increased ethernet activity

#10 2011-03-28 18:56:09

Re: System lock-ups or software resets, with increased ethernet activity

#11 2011-03-29 23:09:57

Re: System lock-ups or software resets, with increased ethernet activity

#12 2011-03-30 00:10:12

Re: System lock-ups or software resets, with increased ethernet activity

#13 2011-03-30 07:22:57

Re: System lock-ups or software resets, with increased ethernet activity

#14 2011-03-30 09:10:39

Re: System lock-ups or software resets, with increased ethernet activity

#15 2011-03-30 16:05:59

Re: System lock-ups or software resets, with increased ethernet activity

#16 2011-04-02 12:40:59

Re: System lock-ups or software resets, with increased ethernet activity

#17 2011-04-09 10:46:59

Re: System lock-ups or software resets, with increased ethernet activity

#18 2011-04-11 12:55:15

Re: System lock-ups or software resets, with increased ethernet activity

#19 2011-04-11 17:59:21

Re: System lock-ups or software resets, with increased ethernet activity

#20 2011-04-11 22:50:19

Re: System lock-ups or software resets, with increased ethernet activity

#21 2011-04-28 20:22:56

Re: System lock-ups or software resets, with increased ethernet activity

#22 2011-04-29 00:03:25

Re: System lock-ups or software resets, with increased ethernet activity

#23 2011-04-29 03:45:30

Re: System lock-ups or software resets, with increased ethernet activity

#24 2011-04-29 11:15:29

Re: System lock-ups or software resets, with increased ethernet activity

#25 2011-04-29 11:56:07

Re: System lock-ups or software resets, with increased ethernet activity

Board footer