Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

stitchrepohlev · 2018-02-21 07:08:37

Every time I can reproduce this by these steps:
1. Power-on switch and notebook
2. Have connected switch to router with DHCP server, ofc to notebook and to I-TEC USB-C docking station i-tec.cz/?t=3&v=437
3. Everything works just fine. Notebook obtain address from DHCP server. it just works...
4. After 10 - 20 minutes notebook will stop networking. DHCP lease persists (I can verify that in NetworkManager) but I can't ping google (8.8.8.8) and neither router itself (10.0.0.1).
5. If I disconnect ethernet from notebook and reconnect it again, notebook won't obtain address. Even after reboot.
6. If I left ethernet connected to notebook or reconnect it (doesn't matter) and disconnect ethernet from docking station, everything start work

Why is it Arch's problem? Because of error log and because everything else works just fine.

Error log:

Feb 19 19:27:14 ArchLinux-VAIO kernel: NETDEV WATCHDOG: enp19s0 (r8169): transmit queue 0 timed out
Feb 19 19:27:14 ArchLinux-VAIO kernel: ------------[ cut here ]------------
Feb 19 19:27:14 ArchLinux-VAIO kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x214/0x220
Feb 19 19:27:14 ArchLinux-VAIO kernel: Modules linked in: rfcomm fuse bnep 8021q mrp nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper hid_g
Feb 19 19:27:14 ArchLinux-VAIO kernel:  r8169  snd_hwdep snd_pcm mei_me snd_timer mei mii snd thermal video soundcore i2c_i801 lpc_ich shpchp battery b
Feb 19 19:27:14 ArchLinux-VAIO kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           O    4.14.19-1-lts #1
Feb 19 19:27:14 ArchLinux-VAIO kernel: Hardware name: Sony Corporation VPCEH3V8E/VAIO, BIOS R0200Z9 04/19/2012
Feb 19 19:27:14 ArchLinux-VAIO kernel: task: ffffffff8b012480 task.stack: ffffffff8b000000
Feb 19 19:27:14 ArchLinux-VAIO kernel: RIP: 0010:dev_watchdog+0x214/0x220
Feb 19 19:27:14 ArchLinux-VAIO kernel: RSP: 0018:ffff8826ff403e88 EFLAGS: 00010282
Feb 19 19:27:14 ArchLinux-VAIO kernel: RAX: 000000000000003c RBX: 0000000000000000 RCX: 0000000000000000
Feb 19 19:27:14 ArchLinux-VAIO kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 0000000000000300
Feb 19 19:27:14 ArchLinux-VAIO kernel: RBP: ffff8826f9a2c45c R08: 00000000000002c6 R09: 0000000000000004
Feb 19 19:27:14 ArchLinux-VAIO kernel: R10: ffff8826ff403cf0 R11: 0000000000000001 R12: ffff8826f7fcd480
Feb 19 19:27:14 ArchLinux-VAIO kernel: R13: 0000000000000000 R14: ffff8826f9a2c000 R15: 0000000000000001
Feb 19 19:27:14 ArchLinux-VAIO kernel: FS:  0000000000000000(0000) GS:ffff8826ff400000(0000) knlGS:0000000000000000
Feb 19 19:27:14 ArchLinux-VAIO kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 19 19:27:14 ArchLinux-VAIO kernel: CR2: 000055da043db058 CR3: 000000001000a004 CR4: 00000000000606f0
Feb 19 19:27:14 ArchLinux-VAIO kernel: Call Trace:
Feb 19 19:27:14 ArchLinux-VAIO kernel:  <IRQ>
Feb 19 19:27:14 ArchLinux-VAIO kernel:  ? qdisc_rcu_free+0x40/0x40
Feb 19 19:27:14 ArchLinux-VAIO kernel:  call_timer_fn+0x30/0x120
Feb 19 19:27:14 ArchLinux-VAIO kernel:  run_timer_softirq+0x3ec/0x440
Feb 19 19:27:14 ArchLinux-VAIO kernel:  ? tick_sched_timer+0x34/0x70
Feb 19 19:27:14 ArchLinux-VAIO kernel:  __do_softirq+0xde/0x2b3
Feb 19 19:27:14 ArchLinux-VAIO kernel:  irq_exit+0xae/0xb0
Feb 19 19:27:14 ArchLinux-VAIO kernel:  smp_apic_timer_interrupt+0x70/0x120
Feb 19 19:27:14 ArchLinux-VAIO kernel:  apic_timer_interrupt+0x98/0xa0
Feb 19 19:27:14 ArchLinux-VAIO kernel:  </IRQ>
Feb 19 19:27:14 ArchLinux-VAIO kernel: RIP: 0010:cpuidle_enter_state+0x9f/0x2e0
Feb 19 19:27:14 ArchLinux-VAIO kernel: RSP: 0018:ffffffff8b003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
Feb 19 19:27:14 ArchLinux-VAIO kernel: RAX: ffff8826ff41f380 RBX: 000004333070d030 RCX: 000000000000001f
Feb 19 19:27:14 ArchLinux-VAIO kernel: RDX: 000004333070d030 RSI: fffffffde515b55e RDI: 0000000000000000
Feb 19 19:27:14 ArchLinux-VAIO kernel: RBP: ffff8826ff427200 R08: 000000000004b3c2 R09: 000000000000e439
Feb 19 19:27:14 ArchLinux-VAIO kernel: R10: ffffffff8b003e70 R11: 0000000000025626 R12: 0000000000000005
Feb 19 19:27:14 ArchLinux-VAIO kernel: R13: ffffffff8b0ac258 R14: 0000000000000000 R15: 000004332bd043da
Feb 19 19:27:14 ArchLinux-VAIO kernel:  do_idle+0x179/0x1d0
Feb 19 19:27:14 ArchLinux-VAIO kernel:  cpu_startup_entry+0x6f/0x80
Feb 19 19:27:14 ArchLinux-VAIO kernel:  start_kernel+0x4b2/0x4d2
Feb 19 19:27:14 ArchLinux-VAIO kernel:  secondary_startup_64+0xa5/0xb0
Feb 19 19:27:14 ArchLinux-VAIO kernel: Code: 63 8e 60 04 00 00 eb 8f 4c 89 f7 c6 05 b9 fc b7 00 01 e8 b0 70 fd ff 89 d9 48 89 c2 4c 89 f6 48 c7 c7 10 
Feb 19 19:27:14 ArchLinux-VAIO kernel: ---[ end trace 5a0f63303d457599 ]---
Feb 19 19:27:14 ArchLinux-VAIO kernel: r8169 0000:13:00.0 enp19s0: link up

If you will need more info, just tell me

Last edited by stitchrepohlev (2018-03-02 12:36:51)

lo1 · 2018-02-22 17:07:42

What do you use to connect? Any network manager or is it a manual setup?

stitchrepohlev · 2018-02-22 18:04:19

NetworkManager in Cinnamon GUI

lo1 · 2018-02-22 21:16:04

Sorry, that was a dumb question you already stated this in the first post. Anyway, my point was I don't know if you can really trust NetworkManager to verify that a dhcp lease has not expired.

It would be easier if you post full logs from the time the connection is started to when it's stopped. When the network works, what's the output of

arp -av

and does it change when the link goes down?

Last edited by lo1 (2018-02-22 21:16:58)

stitchrepohlev · 2018-02-23 20:06:03

[user@ArchLinux-VAIO ~]$ sudo arp -av
router (10.0.0.1) at 6c:3b:6b:42:3a:a4 [ether] on enp19s0
Entries: 1	Skipped: 0	Found: 1

#After while

[user@ArchLinux-VAIO ~]$ sudo arp -av
[sudo] password for user: 
Entries: 0	Skipped: 0	Found: 0
[user@ArchLinux-VAIO ~]$ ifconfig
enp19s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9f71:46e6:5c1a:f5e8  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::cc89:386b:7246:498f  prefixlen 64  scopeid 0x20<link>
        ether f0:bf:97:05:84:36  txqueuelen 1000  (Ethernet)
        RX packets 604981  bytes 780922798 (744.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 306866  bytes 33477730 (31.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 324  bytes 31009 (30.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 324  bytes 31009 (30.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp7s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 42:ff:70:9c:2b:fb  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[user@ArchLinux-VAIO ~]$ ethtool -i enp19s0
driver: r8169
version: 2.3LK-NAPI
firmware-version: rtl_nic/rtl8168e-2.fw
expansion-rom-version: 
bus-info: 0000:13:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[rozarka@ArchLinux-VAIO ~]$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable

Last edited by stitchrepohlev (2018-02-23 21:52:25)

V1del · 2018-02-23 20:38:15

Please wrap output in [ code ] tags.

https://bbs.archlinux.org/help.php#bbcode

Last edited by V1del (2018-02-23 20:38:35)

lo1 · 2018-02-23 21:29:09

Indeed, use code tags, the output is not easily readable.

Anyway:

stitchrepohlev wrote:

#After while

[user@ArchLinux-VAIO ~]$ sudo arp -av
Entries: 0    Skipped: 0    Found: 0

there's no mac address found from your arp cache, hence the dhcp lease expires (probably this USB dongle does other strange things, regardless that it works well with other systems/devices).

I'm not that good with layer 2 connectivity, anyway I think that basically your router passes dhcp offers to your USB interface correctly (they're generated from your laptop with the correct source mac address and sent in broadcast UDP frames, port 68 to 67), but something goes wrong when the renewal is issued. Either you or the router are not receiving the unicast UDP frames.

If I'm right, after stopping network manager, shutting down the interface, connect/disconnect the usb dongle and manually connecting (no network manager) specifing a greater lease time you should see your network timing out less frequently (assuming that your dhcp server allows you to set a greater lease time, this is done with the command `dhcpcd -l`).

If it works, we'll figure something better than a workaround. If it doesn't work, we're barking at the wrong tree.

stitchrepohlev · 2018-02-23 21:58:04

You got it bad.

USB dongle is other device which is independent from VAIO (ntb which has problem).

VAIO has built in ethernet card:

[user@ArchLinux-VAIO ~]$ ethtool -i enp19s0
driver: r8169
version: 2.3LK-NAPI
firmware-version: rtl_nic/rtl8168e-2.fw
expansion-rom-version: 
bus-info: 0000:13:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

dongle is just other device connected to same switch. Also all works while dongle is used (like is connected to MacBook Pro for example) but when it isn't, it cases problem. But only for VAIO with ArchLinux... Nothing else has problem... Dongle has PowerDelivery function which means that after disconnecting it from MBP it still working, has IP lease, etc...

lo1 · 2018-02-24 09:08:40

dongle is just other device connected to same switch

I see, this wasn't really clear from your first post. Anyway, I think this is the same issue: something is "stealing" your dhcp renewals or they're just being sent to the wrong device.

Dongle has PowerDelivery function which means that after disconnecting it from MBP it still working, has IP lease, etc...

Which address do you get and which one does your USB dongle gets? What's the address of your dhcp server?

Check for the mac address in your network (every point to point links) and if you can use something like wireshark to follow the frames when you connect/disconnect the dongle.

Last edited by lo1 (2018-02-24 09:23:28)

stitchrepohlev · 2018-02-24 10:42:56

Dongle has MAC Address 00:E0:4C:31:51:92 and has static IP 10.0.0.2

I just found strange thing that VAIO obtain 2 leases for 1 ethernet card

First MAC Address is F0:BF:97:05:84:36 and has static IP 10.0.0.12
Second MAC Address F0:BF:97:05:84:36 and dynamic IP lease 10.0.0.13

yes, both VAIO MAC Address are same, only client ID differs

Last edited by stitchrepohlev (2018-02-24 18:51:45)

lo1 · 2018-02-24 10:53:59

The dongle and the VAIO interfaces have the same mac address (why on earth?), plus your VAIO is using both static and dynamic IP addressing one the same interface?

Now it's no surprise that this is not working. How did you setup your network on Arch?

stitchrepohlev · 2018-02-24 18:55:34

I am really sorry... I didn't check mine reply before sending....

Dongle's and VAIO's MAC address AREN'T same. I edited original post but to be sure:

Dongle's MAC Address: 00:E0:4C:31:51:92

VAIO's 2 leases MAC Address: F0:BF:97:05:84:36

VAIO has one static lease on router because I set it up it before. It didn't have second lease for same interface before or I just didn't notice it.

Last edited by stitchrepohlev (2018-02-24 19:32:53)

lo1 · 2018-02-24 20:34:34

I am really sorry... I didn't check mine reply before sending

Nevermind, is not a big deal

Anyway, just do as I suggested in post #6: for the moment forget about dhcp and disable any network manager, shutdown the interface, then using the `ip` commands flush your interface configuration, restart the interface, assign a manual ip address and a route to your gateway.

stitchrepohlev · 2018-02-28 21:22:21

I did:

pacman -R networkmanager

restarted OS and tried it but it didn't work... same as before

lo1 · 2018-03-01 19:42:47

Removing networkmanager was a bit of an overkill, and I'm also not sure if it may leave some services hanging around.

EDIT: the easy and sane way is to disable a service, not removing the program which is called by such service.

tried it but it didn't work... same as before

Please don't do this, post the exact step by step terminal input/output and logs if relevant.

Last edited by lo1 (2018-03-01 19:47:41)

stitchrepohlev · 2018-03-02 12:47:52

I did:

pacman -S networkmanager

And it didn't work. Same crash as I posted in first post: https://bbs.archlinux.org/viewtopic.php … 9#p1769439

Then I did:

systemctl disable NetworkManager

Also didn't work. Same crash as I posted in first post: https://bbs.archlinux.org/viewtopic.php … 9#p1769439

I verified that windows ntb and macOS ntb works while connected to same switch as non-working Arch. I mentioned Arch because it seems to bit only Arch's problem. I can't verify, if other linux distros work but I tried one desktop Arch (the one I told you about in first post) and I also verified that Raspberry Pi with ArmArch also doesn't work. Exactly same problem as normal 64-bit Arch.

Another interesting thing is Arch's client id... Every system (windows, iOS, macOS, ...) uses 7 format (I don't know how to call it) like 1:aa:bb:cc:dd:ee:ff and every time starts with `1` but Arch uses 19 smt format and eveyr time starts with `ff`. Screenshot:

Screenshot

I have 3 arch machines in my local network.

First two rows is ArchLinux 64-bit ntb
Third and Fourth are two Raspberry Pi-s with ArmArchLinux

lo1 · 2018-03-02 17:12:53

I really have no clue, sorry. Do you know if this happened after a specific e.g. kernel release, or it's always been an issue with arch* + your network setup?

stitchrepohlev · 2018-03-03 22:05:07

This problem started when I bought USB-C docking station. (Problem with infunctional internet).

The weird client ID... I believe that it is since beggining. You doesn't have this problem?

My network:

WAN is Mikrotik Router with DHCP Server

Both switches are same brand and type. Both are D-Link DGS-108/E

Windows Machine works while ArchLinux and ArchArm doesn't because macOS is disconnected from it's docking station. I believe that ArchArm Pi on Switch1 works. Only the one on Switch2 doesn't because it is same switch as USB-C docking station

Network Map

Last edited by stitchrepohlev (2018-03-03 22:06:27)

lo1 · 2018-03-03 22:44:43

Nope, never had this nor any other problem (using rtl8723).

Also, my client id (duid, according to RFC 3315 is similar to yours, with the last 48 bits being the MAC-ADDRESS of my machine. Though this is scratching my mind too.

Can you try to backup /var/lib/dhcpcd/duid and change any value before your mac address? (be sure that the end result is a 128 bit value, so change just one number/letter to another within [0-9][a-f])

stitchrepohlev · 2018-03-05 20:50:06

I can reproduce it also in OSMC with kernel 4.9.29
Arch has kernel 4.15.6 now

I'll try some old system with old kernel like 3 or 4.1, ...

I didn't try your suggestion yet, sorry, I'll do try it asap

Last edited by stitchrepohlev (2018-03-05 20:50:19)

lo1 · 2018-03-10 21:59:35

Ok, I finally had a look at RFC 3315. You were right: the DUID is invalid. Either try to modify it so that it fits the standard as per section 9.1 of the related RFC, or try to delete/move it (preferably before attempting a connection) in order to force a regeneration.

Arch Linux

#1 2018-02-21 07:08:37

Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#2 2018-02-22 17:07:42

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#3 2018-02-22 18:04:19

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#4 2018-02-22 21:16:04

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#5 2018-02-23 20:06:03

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#6 2018-02-23 20:38:15

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#7 2018-02-23 21:29:09

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#8 2018-02-23 21:58:04

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#9 2018-02-24 09:08:40

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#10 2018-02-24 10:42:56

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#11 2018-02-24 10:53:59

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#12 2018-02-24 18:55:34

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#13 2018-02-24 20:34:34

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#14 2018-02-28 21:22:21

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#15 2018-03-01 19:42:47

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#16 2018-03-02 12:47:52

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#17 2018-03-02 17:12:53

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#18 2018-03-03 22:05:07

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#19 2018-03-03 22:44:43

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#20 2018-03-05 20:50:06

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

#21 2018-03-10 21:59:35

Re: Network crashes with WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c

Board footer