You are not logged in.

Pages: 1

#1 2019-07-22 00:11:39

nc99
Member
Registered: 2019-07-21
Posts: 3

[SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

Hi, I've recently installed Arch Linux on a new computer I put together. Everything works great, except after a few minutes of using Wireguard the system hangs.

I can consistently reproduce this issue on my machine (but I haven't tried it on any others since I don't have any other computers running Arch Linux at the moment). I am connected on ethernet, and normal networking works fine, and I have tried both NetworkManager and systemd-networkd. When I start up Wireguard, either through systemd-networkd native support or with wireguard's systemd service (sudo systemctl start wg-quick@intranet) ("intranet" is the name of my Wireguard network) (configured as per the wiki's instructions), it works perfectly fine for a couple of minutes. I can even ping other hosts on my Wireguard network. However, after a couple minutes, the CPU load on a single CPU core spikes to 100% (I watch htop). Shortly thereafter, all pings to the Wireguard network fail. Normal internet keeps working for a bit, but then that also fails (all pings and network-related commands hang). I then have to forcibly restart my computer by holding the power button to be able to use anything network related again.

Based on the output of dmesg (below), I'm thinking this may be a bug with the Wireguard kernel module, but I'm not sure. I've looked at these forums and online, and have not found anything similar related to Wireguard. It could also be poor support for my CPU since it is relatively new (Ryzen 9 3900X). If anyone knows how to resolve this issue, or next steps I should take to troubleshoot, I'd really appreciate your advice.

Thank you,
Nicolas

-----

Here is the output of "ip a", showing all network interfaces as I'd expect:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b4:2e:99:3d:2d:ad brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.224/24 brd 192.168.1.255 scope global dynamic noprefixroute enp3s0
       valid_lft 42603sec preferred_lft 42603sec
    inet6 fded:3c4d:d569::cba/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fded:3c4d:d569:0:741a:ecd6:9bcb:ca86/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::68d:8bf2:321b:aec4/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: wlp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 8a:66:06:4a:7c:9a brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 7e:b9:69:56:e5:11 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: intranet: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.0.9.10/24 scope global intranet
       valid_lft forever preferred_lft forever

dmesg notices the issue:

[  545.876435] wireguard: WireGuard 0.0.20190702 loaded. See www.wireguard.com for information.
[  545.876436] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[  708.146753] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u64:5:808]
[  708.146755] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel fuse xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc edac_mce_amd snd_hda_codec_hdmi arc4 nvidia_drm(POE) nvidia_modeset(POE) nls_iso8859_1 nls_cp437 vfat fat kvm iwlmvm nvidia(POE) irqbypass mac80211 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio btusb btrtl crct10dif_pclmul btbcm snd_hda_intel btintel crc32_pclmul ghash_clmulni_intel hid_uclogic joydev mousedev bluetooth wmi_bmof input_leds iwlwifi snd_hda_codec drm_kms_helper snd_hda_core ecdh_generic ecc snd_hwdep crc16 drm cfg80211 aesni_intel snd_pcm aes_x86_64 agpgart crypto_simd cryptd ipmi_devintf igb glue_helper snd_timer ipmi_msghandler snd syscopyarea sp5100_tco sysfillrect i2c_piix4 ccp sysimgblt pcspkr fb_sys_fops evdev i2c_algo_bit soundcore rfkill rng_core dca mac_hid pinctrl_amd wmi pcc_cpufreq
[  708.146779]  acpi_cpufreq sg crypto_user ip_tables x_tables btrfs libcrc32c crc32c_generic xor raid6_pq sd_mod hid_generic usbhid hid ahci libahci libata xhci_pci crc32c_intel scsi_mod xhci_hcd
[  708.146786] CPU: 2 PID: 808 Comm: kworker/u64:5 Tainted: P           OE     5.2.1-arch1-1-ARCH #1
[  708.146787] Hardware name: Gigabyte Technology Co., Ltd. X570 I AORUS PRO WIFI/X570 I AORUS PRO WIFI, BIOS F3 06/13/2019
[  708.146792] Workqueue: wg-kex-intranet wg_packet_handshake_send_worker [wireguard]
[  708.146795] RIP: 0010:get_random_u32+0x21/0xa0
[  708.146796] Code: b8 01 00 00 00 eb cc 66 90 0f 1f 44 00 00 41 54 55 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 <72> 5f 83 ea 01 75 f6 48 8b 74 24 20 48 c7 c2 20 84 1c b5 48 c7 c7
[  708.146797] RSP: 0018:ffffa543c9123ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff13
[  708.146797] RAX: 00000000ffffffff RBX: ffffa05d720b42c8 RCX: 0000000000000000
[  708.146798] RDX: 000000000000000a RSI: 0000000000000201 RDI: ffffffffc0ef6297
[  708.146798] RBP: ffffa05d3ac60000 R08: ffffffffc0f1eea0 R09: ffffa543c9123c58
[  708.146798] R10: 0000000000000000 R11: 073b28980328607f R12: ffffa05d3ac70000
[  708.146799] R13: ffffa05d720b42d0 R14: ffffa543c9123d30 R15: ffffa05d720b43f0
[  708.146799] FS:  0000000000000000(0000) GS:ffffa05dbe680000(0000) knlGS:0000000000000000
[  708.146800] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  708.146800] CR2: 00007efbdec04000 CR3: 00000007ebd58000 CR4: 0000000000340ee0
[  708.146801] Call Trace:
[  708.146804]  ? wg_index_hashtable_insert+0x6f/0x110 [wireguard]
[  708.146807]  wg_index_hashtable_insert+0x5c/0x110 [wireguard]
[  708.146809]  wg_noise_handshake_create_initiation+0x2d6/0x300 [wireguard]
[  708.146812]  wg_packet_send_handshake_initiation+0xc9/0x160 [wireguard]
[  708.146814]  wg_packet_handshake_send_worker+0x18/0x30 [wireguard]
[  708.146816]  process_one_work+0x1d1/0x3e0
[  708.146817]  worker_thread+0x4a/0x3d0
[  708.146818]  kthread+0xfd/0x130
[  708.146819]  ? process_one_work+0x3e0/0x3e0
[  708.146820]  ? kthread_park+0x90/0x90
[  708.146822]  ret_from_fork+0x22/0x40
[  736.146757] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u64:5:808]
[  736.146759] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel fuse xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc edac_mce_amd snd_hda_codec_hdmi arc4 nvidia_drm(POE) nvidia_modeset(POE) nls_iso8859_1 nls_cp437 vfat fat kvm iwlmvm nvidia(POE) irqbypass mac80211 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio btusb btrtl crct10dif_pclmul btbcm snd_hda_intel btintel crc32_pclmul ghash_clmulni_intel hid_uclogic joydev mousedev bluetooth wmi_bmof input_leds iwlwifi snd_hda_codec drm_kms_helper snd_hda_core ecdh_generic ecc snd_hwdep crc16 drm cfg80211 aesni_intel snd_pcm aes_x86_64 agpgart crypto_simd cryptd ipmi_devintf igb glue_helper snd_timer ipmi_msghandler snd syscopyarea sp5100_tco sysfillrect i2c_piix4 ccp sysimgblt pcspkr fb_sys_fops evdev i2c_algo_bit soundcore rfkill rng_core dca mac_hid pinctrl_amd wmi pcc_cpufreq
[  736.146783]  acpi_cpufreq sg crypto_user ip_tables x_tables btrfs libcrc32c crc32c_generic xor raid6_pq sd_mod hid_generic usbhid hid ahci libahci libata xhci_pci crc32c_intel scsi_mod xhci_hcd
[  736.146789] CPU: 2 PID: 808 Comm: kworker/u64:5 Tainted: P           OEL    5.2.1-arch1-1-ARCH #1
[  736.146790] Hardware name: Gigabyte Technology Co., Ltd. X570 I AORUS PRO WIFI/X570 I AORUS PRO WIFI, BIOS F3 06/13/2019
[  736.146796] Workqueue: wg-kex-intranet wg_packet_handshake_send_worker [wireguard]
[  736.146799] RIP: 0010:get_random_u32+0x21/0xa0
[  736.146800] Code: b8 01 00 00 00 eb cc 66 90 0f 1f 44 00 00 41 54 55 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 <72> 5f 83 ea 01 75 f6 48 8b 74 24 20 48 c7 c2 20 84 1c b5 48 c7 c7
[  736.146800] RSP: 0018:ffffa543c9123ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff13
[  736.146801] RAX: 00000000ffffffff RBX: ffffa05d720b42c8 RCX: 0000000000000000
[  736.146802] RDX: 000000000000000a RSI: 0000000000000201 RDI: ffffffffc0ef6297
[  736.146802] RBP: ffffa05d3ac60000 R08: ffffffffc0f1eea0 R09: ffffa543c9123c58
[  736.146802] R10: 0000000000000000 R11: 073b28980328607f R12: ffffa05d3ac70000
[  736.146803] R13: ffffa05d720b42d0 R14: ffffa543c9123d30 R15: ffffa05d720b43f0
[  736.146803] FS:  0000000000000000(0000) GS:ffffa05dbe680000(0000) knlGS:0000000000000000
[  736.146804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  736.146804] CR2: 00007efbdec04000 CR3: 00000007ebd58000 CR4: 0000000000340ee0
[  736.146805] Call Trace:
[  736.146809]  ? wg_index_hashtable_insert+0x7d/0x110 [wireguard]
[  736.146811]  wg_index_hashtable_insert+0x5c/0x110 [wireguard]
[  736.146813]  wg_noise_handshake_create_initiation+0x2d6/0x300 [wireguard]
[  736.146816]  wg_packet_send_handshake_initiation+0xc9/0x160 [wireguard]
[  736.146818]  wg_packet_handshake_send_worker+0x18/0x30 [wireguard]
[  736.146820]  process_one_work+0x1d1/0x3e0
[  736.146821]  worker_thread+0x4a/0x3d0
[  736.146822]  kthread+0xfd/0x130
[  736.146823]  ? process_one_work+0x3e0/0x3e0
[  736.146824]  ? kthread_park+0x90/0x90
[  736.146826]  ret_from_fork+0x22/0x40
[  740.753351] rcu: INFO: rcu_preempt self-detected stall on CPU
[  740.753355] rcu:     2-....: (8 GPs behind) idle=bd2/1/0x4000000000000002 softirq=1024/1024 fqs=5999 last_accelerate: bdca/048c, Nonlazy posted: .LD
[  740.753358]  (t=18000 jiffies g=9985 q=125494)
[  740.753359] NMI backtrace for cpu 2
[  740.753361] CPU: 2 PID: 808 Comm: kworker/u64:5 Tainted: P           OEL    5.2.1-arch1-1-ARCH #1
[  740.753361] Hardware name: Gigabyte Technology Co., Ltd. X570 I AORUS PRO WIFI/X570 I AORUS PRO WIFI, BIOS F3 06/13/2019
[  740.753368] Workqueue: wg-kex-intranet wg_packet_handshake_send_worker [wireguard]
[  740.753370] Call Trace:
[  740.753373]  <IRQ>
[  740.753377]  dump_stack+0x5c/0x78
[  740.753379]  nmi_cpu_backtrace.cold+0x13/0x51
[  740.753381]  ? lapic_can_unplug_cpu.cold+0x44/0x44
[  740.753383]  nmi_trigger_cpumask_backtrace+0x10e/0x11f
[  740.753385]  rcu_dump_cpu_stacks+0x92/0xc0
[  740.753387]  rcu_sched_clock_irq.cold+0x25c/0x4fb
[  740.753389]  update_process_times+0x29/0x60
[  740.753390]  tick_sched_handle.isra.0+0x22/0x60
[  740.753392]  tick_sched_timer+0x3b/0x80
[  740.753393]  ? tick_sched_handle.isra.0+0x60/0x60
[  740.753394]  __hrtimer_run_queues+0xfd/0x2c0
[  740.753396]  hrtimer_interrupt+0x118/0x240
[  740.753397]  ? sched_clock+0x5/0x10
[  740.753399]  smp_apic_timer_interrupt+0x84/0x180
[  740.753401]  apic_timer_interrupt+0xf/0x20
[  740.753402]  </IRQ>
[  740.753404] RIP: 0010:get_random_u32+0x21/0xa0
[  740.753405] Code: b8 01 00 00 00 eb cc 66 90 0f 1f 44 00 00 41 54 55 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 <72> 5f 83 ea 01 75 f6 48 8b 74 24 20 48 c7 c2 20 84 1c b5 48 c7 c7
[  740.753406] RSP: 0018:ffffa543c9123ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff13
[  740.753407] RAX: 00000000ffffffff RBX: ffffa05d720b42c8 RCX: 0000000000000000
[  740.753407] RDX: 000000000000000a RSI: 0000000000000201 RDI: ffffffffc0ef6297
[  740.753408] RBP: ffffa05d3ac60000 R08: ffffffffc0f1eea0 R09: ffffa543c9123c58
[  740.753408] R10: 0000000000000000 R11: 073b28980328607f R12: ffffa05d3ac70000
[  740.753409] R13: ffffa05d720b42d0 R14: ffffa543c9123d30 R15: ffffa05d720b43f0
[  740.753413]  ? wg_index_hashtable_insert+0x57/0x110 [wireguard]
[  740.753416]  ? wg_index_hashtable_insert+0x7d/0x110 [wireguard]
[  740.753418]  wg_index_hashtable_insert+0x5c/0x110 [wireguard]
[  740.753421]  wg_noise_handshake_create_initiation+0x2d6/0x300 [wireguard]
[  740.753425]  wg_packet_send_handshake_initiation+0xc9/0x160 [wireguard]
[  740.753428]  wg_packet_handshake_send_worker+0x18/0x30 [wireguard]
[  740.753430]  process_one_work+0x1d1/0x3e0
[  740.753431]  worker_thread+0x4a/0x3d0
[  740.753433]  kthread+0xfd/0x130
[  740.753434]  ? process_one_work+0x3e0/0x3e0
[  740.753435]  ? kthread_park+0x90/0x90
[  740.753436]  ret_from_fork+0x22/0x40
[  768.146760] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u64:5:808]
[  768.146762] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel fuse xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc edac_mce_amd snd_hda_codec_hdmi arc4 nvidia_drm(POE) nvidia_modeset(POE) nls_iso8859_1 nls_cp437 vfat fat kvm iwlmvm nvidia(POE) irqbypass mac80211 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio btusb btrtl crct10dif_pclmul btbcm snd_hda_intel btintel crc32_pclmul ghash_clmulni_intel hid_uclogic joydev mousedev bluetooth wmi_bmof input_leds iwlwifi snd_hda_codec drm_kms_helper snd_hda_core ecdh_generic ecc snd_hwdep crc16 drm cfg80211 aesni_intel snd_pcm aes_x86_64 agpgart crypto_simd cryptd ipmi_devintf igb glue_helper snd_timer ipmi_msghandler snd syscopyarea sp5100_tco sysfillrect i2c_piix4 ccp sysimgblt pcspkr fb_sys_fops evdev i2c_algo_bit soundcore rfkill rng_core dca mac_hid pinctrl_amd wmi pcc_cpufreq
[  768.146785]  acpi_cpufreq sg crypto_user ip_tables x_tables btrfs libcrc32c crc32c_generic xor raid6_pq sd_mod hid_generic usbhid hid ahci libahci libata xhci_pci crc32c_intel scsi_mod xhci_hcd
[  768.146791] CPU: 2 PID: 808 Comm: kworker/u64:5 Tainted: P           OEL    5.2.1-arch1-1-ARCH #1
[  768.146791] Hardware name: Gigabyte Technology Co., Ltd. X570 I AORUS PRO WIFI/X570 I AORUS PRO WIFI, BIOS F3 06/13/2019
[  768.146797] Workqueue: wg-kex-intranet wg_packet_handshake_send_worker [wireguard]
[  768.146800] RIP: 0010:get_random_u32+0x21/0xa0
[  768.146801] Code: b8 01 00 00 00 eb cc 66 90 0f 1f 44 00 00 41 54 55 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 <72> 5f 83 ea 01 75 f6 48 8b 74 24 20 48 c7 c2 20 84 1c b5 48 c7 c7
[  768.146801] RSP: 0018:ffffa543c9123ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff13
[  768.146802] RAX: 00000000ffffffff RBX: ffffa05d720b42c8 RCX: 0000000000000000
[  768.146802] RDX: 000000000000000a RSI: 0000000000000201 RDI: ffffffffc0ef6297
[  768.146802] RBP: ffffa05d3ac60000 R08: ffffffffc0f1eea0 R09: ffffa543c9123c58
[  768.146803] R10: 0000000000000000 R11: 073b28980328607f R12: ffffa05d3ac70000
[  768.146803] R13: ffffa05d720b42d0 R14: ffffa543c9123d30 R15: ffffa05d720b43f0
[  768.146804] FS:  0000000000000000(0000) GS:ffffa05dbe680000(0000) knlGS:0000000000000000
[  768.146804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  768.146805] CR2: 00007efbdec04000 CR3: 00000007ebd58000 CR4: 0000000000340ee0
[  768.146805] Call Trace:
[  768.146808]  ? wg_index_hashtable_insert+0x6f/0x110 [wireguard]
[  768.146810]  wg_index_hashtable_insert+0x5c/0x110 [wireguard]
[  768.146813]  wg_noise_handshake_create_initiation+0x2d6/0x300 [wireguard]
[  768.146815]  wg_packet_send_handshake_initiation+0xc9/0x160 [wireguard]
[  768.146817]  wg_packet_handshake_send_worker+0x18/0x30 [wireguard]
[  768.146819]  process_one_work+0x1d1/0x3e0
[  768.146820]  worker_thread+0x4a/0x3d0
[  768.146821]  kthread+0xfd/0x130
[  768.146822]  ? process_one_work+0x3e0/0x3e0
[  768.146823]  ? kthread_park+0x90/0x90
[  768.146824]  ret_from_fork+0x22/0x40

Last edited by nc99 (2019-07-22 09:45:33)

Offline

#2 2019-07-22 07:17:39

dpzmick
Member
Registered: 2013-07-19
Posts: 3

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

I too have noticed this issue, here's a giant dump of some dmesg output around the time of the issue.

This might be untrue, but I've also had similar issues with a virtual machine running on my host.

My machine has a new ryzen 3800x and a relatively new motherboard. Looks like OP might have the same chipset, so probably has a similar generation of CPU.

[ 2367.181765] wireguard: WireGuard 0.0.20190702 loaded. See www.wireguard.com for information.
[ 2367.181766] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[ 2528.243393] ------------[ cut here ]------------
[ 2528.243407] NETDEV WATCHDOG: enp4s0 (igb): transmit queue 0 timed out
[ 2528.243422] WARNING: CPU: 10 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x255/0x260
[ 2528.243423] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel nct6775 hwmon_vid jc42 eeprom cfg80211 8021q garp mrp stp llc edac_mce_amd ax25 snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) nls_iso8859_1 nls_cp437 vfat fat nvidia(POE) kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul drm_kms_helper ledtrig_audio ghash_clmulni_intel snd_usb_audio drm snd_hda_intel input_leds igb snd_hda_codec snd_usbmidi_lib aesni_intel snd_rawmidi mousedev snd_seq_device eeepc_wmi snd_hda_core asus_wmi media sparse_keymap aes_x86_64 rfkill snd_hwdep crypto_simd sp5100_tco cryptd wmi_bmof mxm_wmi glue_helper snd_pcm agpgart pcspkr i2c_piix4 i2c_algo_bit ipmi_devintf snd_timer dca ipmi_msghandler snd syscopyarea ccp sysfillrect sysimgblt fb_sys_fops soundcore rng_core wmi evdev pinctrl_amd pcc_cpufreq mac_hid acpi_cpufreq vboxnetflt(OE) vboxnetadp(OE) vboxpci(OE) vboxdrv(OE) ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 sd_mod hid_generic
[ 2528.243481]  usbhid hid ahci libahci libata xhci_pci crc32c_intel xhci_hcd scsi_mod [last unloaded: cpuid]
[ 2528.243489] CPU: 10 PID: 0 Comm: swapper/10 Tainted: P           OE     5.2.1-arch1-1-ARCH #1
[ 2528.243491] Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 0808 07/08/2019
[ 2528.243493] RIP: 0010:dev_watchdog+0x255/0x260
[ 2528.243495] Code: 04 63 83 ff eb 8a 4c 89 f7 c6 05 43 8d b4 00 01 e8 a0 04 fb ff 44 89 e9 4c 89 f6 48 c7 c7 f8 b7 d5 a8 48 89 c2 e8 6a 1c 8c ff <0f> 0b e9 68 ff ff ff 0f 1f 40 00 0f 1f 44 00 00 41 54 49 89 d4 55
[ 2528.243496] RSP: 0018:ffffb717035acdf8 EFLAGS: 00010282
[ 2528.243498] RAX: 0000000000000000 RBX: ffff9ec634b818c0 RCX: 0000000000000000
[ 2528.243499] RDX: 0000000000040400 RSI: 00000000000000f6 RDI: 0000000000000001
[ 2528.243500] RBP: ffff9ec62c28c480 R08: 00000000000003fc R09: 0000000000000001
[ 2528.243501] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9ec62c28c45c
[ 2528.243502] R13: 0000000000000000 R14: ffff9ec62c28c000 R15: ffff9ec634b81940
[ 2528.243503] FS:  0000000000000000(0000) GS:ffff9ec63ea80000(0000) knlGS:0000000000000000
[ 2528.243504] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2528.243506] CR2: 00007efbea368000 CR3: 0000000797d1a000 CR4: 0000000000340ee0
[ 2528.243507] Call Trace:
[ 2528.243509]  <IRQ>
[ 2528.243513]  ? qdisc_put_unlocked+0x30/0x30
[ 2528.243517]  call_timer_fn+0x30/0x170
[ 2528.243518]  ? qdisc_put_unlocked+0x30/0x30
[ 2528.243521]  expire_timers+0xa9/0x120
[ 2528.243523]  run_timer_softirq+0xbd/0x1b0
[ 2528.243528]  __do_softirq+0x119/0x34f
[ 2528.243532]  irq_exit+0xd4/0xf0
[ 2528.243535]  smp_apic_timer_interrupt+0x8e/0x180
[ 2528.243537]  apic_timer_interrupt+0xf/0x20
[ 2528.243539]  </IRQ>
[ 2528.243542] RIP: 0010:cpuidle_enter_state+0xc4/0x480
[ 2528.243543] Code: e8 41 63 9a ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 93 03 00 00 31 ff e8 e3 fe a0 ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 88 be 02 00 00 49 63 cc 4c 2b 6c 24 10 48 8d 04 49 48
[ 2528.243544] RSP: 0018:ffffb717032a3e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 2528.243545] RAX: ffff9ec63ea80000 RBX: ffffffffa8ec0d40 RCX: 000000000000001f
[ 2528.243546] RDX: 0000000000000000 RSI: 0000000020e1d712 RDI: 0000000000000000
[ 2528.243547] RBP: ffff9ec636bcd000 R08: 0000024ca70c6e4a R09: 000000007fffffff
[ 2528.243548] R10: ffff9ec63eaa92c4 R11: ffff9ec63eaa92a4 R12: 0000000000000002
[ 2528.243549] R13: 0000024ca70c6e4a R14: 0000000000000002 R15: ffff9ec63acd9e40
[ 2528.243552]  ? cpuidle_enter_state+0x9f/0x480
[ 2528.243554]  cpuidle_enter+0x29/0x40
[ 2528.243557]  do_idle+0x1ec/0x270
[ 2528.243559]  cpu_startup_entry+0x19/0x20
[ 2528.243562]  start_secondary+0x19f/0x1f0
[ 2528.243565]  secondary_startup_64+0xa4/0xb0
[ 2528.243568] ---[ end trace 2cbf1a6ccb848714 ]---
[ 2528.243783] igb 0000:04:00.0 enp4s0: Reset adapter
[ 2528.586502] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kworker/u64:1:1109]
[ 2528.586504] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel nct6775 hwmon_vid jc42 eeprom cfg80211 8021q garp mrp stp llc edac_mce_amd ax25 snd_hda_codec_hdmi nvidia_drm(POE) nvidia_modeset(POE) nls_iso8859_1 nls_cp437 vfat fat nvidia(POE) kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul drm_kms_helper ledtrig_audio ghash_clmulni_intel snd_usb_audio drm snd_hda_intel input_leds igb snd_hda_codec snd_usbmidi_lib aesni_intel snd_rawmidi mousedev snd_seq_device eeepc_wmi snd_hda_core asus_wmi media sparse_keymap aes_x86_64 rfkill snd_hwdep crypto_simd sp5100_tco cryptd wmi_bmof mxm_wmi glue_helper snd_pcm agpgart pcspkr i2c_piix4 i2c_algo_bit ipmi_devintf snd_timer dca ipmi_msghandler snd syscopyarea ccp sysfillrect sysimgblt fb_sys_fops soundcore rng_core wmi evdev pinctrl_amd pcc_cpufreq mac_hid acpi_cpufreq vboxnetflt(OE) vboxnetadp(OE) vboxpci(OE) vboxdrv(OE) ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 sd_mod hid_generic
[ 2528.586530]  usbhid hid ahci libahci libata xhci_pci crc32c_intel xhci_hcd scsi_mod [last unloaded: cpuid]
[ 2528.586535] CPU: 13 PID: 1109 Comm: kworker/u64:1 Tainted: P        W  OE     5.2.1-arch1-1-ARCH #1
[ 2528.586535] Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 0808 07/08/2019
[ 2528.586541] Workqueue: wg-kex-wg0 wg_packet_handshake_send_worker [wireguard]
[ 2528.586545] RIP: 0010:get_random_u32+0x21/0xa0
[ 2528.586546] Code: b8 01 00 00 00 eb cc 66 90 0f 1f 44 00 00 41 54 55 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 <72> 5f 83 ea 01 75 f6 48 8b 74 24 20 48 c7 c2 20 84 5c a9 48 c7 c7
[ 2528.586546] RSP: 0018:ffffb71708c47ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff13
[ 2528.586547] RAX: 00000000ffffffff RBX: ffff9ec5e5714ac8 RCX: 0000000000000000
[ 2528.586547] RDX: 000000000000000a RSI: 0000000000000201 RDI: ffffffffc0b3d297
[ 2528.586548] RBP: ffff9ec5b6d20000 R08: ffffffffc0b65ea0 R09: ffffb71708c47c58
[ 2528.586548] R10: 0000000000000000 R11: 0376860c0ac8b9c8 R12: ffff9ec5b6d30000
[ 2528.586548] R13: ffff9ec5e5714ad0 R14: ffffb71708c47d30 R15: ffff9ec5e5714bf0
[ 2528.586549] FS:  0000000000000000(0000) GS:ffff9ec63eb40000(0000) knlGS:0000000000000000
[ 2528.586549] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2528.586550] CR2: 00007efbc848f000 CR3: 00000005dbc0a000 CR4: 0000000000340ee0
[ 2528.586550] Call Trace:
[ 2528.586554]  ? wg_index_hashtable_insert+0x7d/0x110 [wireguard]
[ 2528.586556]  wg_index_hashtable_insert+0x5c/0x110 [wireguard]
[ 2528.586558]  wg_noise_handshake_create_initiation+0x2d6/0x300 [wireguard]
[ 2528.586561]  wg_packet_send_handshake_initiation+0xc9/0x160 [wireguard]
[ 2528.586563]  wg_packet_handshake_send_worker+0x18/0x30 [wireguard]
[ 2528.586565]  process_one_work+0x1d1/0x3e0
[ 2528.586566]  worker_thread+0x4a/0x3d0
[ 2528.586567]  kthread+0xfd/0x130
[ 2528.586568]  ? process_one_work+0x3e0/0x3e0
[ 2528.586569]  ? kthread_park+0x90/0x90
[ 2528.586571]  ret_from_fork+0x22/0x40
[ 2530.376852] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX



 2702.963313] INFO: task kworker/1:2:127 blocked for more than 122 seconds.
[ 2702.963315]       Tainted: P        W  OEL    5.2.1-arch1-1-ARCH #1
[ 2702.963316] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2702.963317] kworker/1:2     D    0   127      2 0x80004000
[ 2702.963329] Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker [wireguard]
[ 2702.963330] Call Trace:
[ 2702.963338]  ? __schedule+0x27f/0x6d0
[ 2702.963340]  schedule+0x3d/0xc0
[ 2702.963342]  rwsem_down_read_failed+0x109/0x1b0
[ 2702.963347]  wg_noise_handshake_consume_initiation+0x38b/0x530 [wireguard]
[ 2702.963353]  wg_receive_handshake_packet+0x13d/0x4e0 [wireguard]
[ 2702.963355]  ? __switch_to_asm+0x40/0x70
[ 2702.963356]  ? __switch_to_asm+0x34/0x70
[ 2702.963361]  wg_packet_handshake_receive_worker+0x25/0x50 [wireguard]
[ 2702.963364]  process_one_work+0x1d1/0x3e0
[ 2702.963366]  worker_thread+0x4a/0x3d0
[ 2702.963368]  kthread+0xfd/0x130
[ 2702.963370]  ? process_one_work+0x3e0/0x3e0
[ 2702.963371]  ? kthread_park+0x90/0x90
[ 2702.963373]  ret_from_fork+0x22/0x40
[ 2702.963377] INFO: task kworker/3:1:144 blocked for more than 122 seconds.
[ 2702.963378]       Tainted: P        W  OEL    5.2.1-arch1-1-ARCH #1
[ 2702.963378] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2702.963379] kworker/3:1     D    0   144      2 0x80004000
[ 2702.963384] Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker [wireguard]
[ 2702.963384] Call Trace:
[ 2702.963387]  ? __schedule+0x27f/0x6d0
[ 2702.963389]  schedule+0x3d/0xc0
[ 2702.963391]  rwsem_down_read_failed+0x109/0x1b0
[ 2702.963395]  wg_noise_handshake_consume_initiation+0x38b/0x530 [wireguard]
[ 2702.963400]  wg_receive_handshake_packet+0x13d/0x4e0 [wireguard]
[ 2702.963401]  ? __switch_to_asm+0x40/0x70
[ 2702.963402]  ? __switch_to_asm+0x34/0x70
[ 2702.963407]  wg_packet_handshake_receive_worker+0x25/0x50 [wireguard]
[ 2702.963408]  process_one_work+0x1d1/0x3e0
[ 2702.963410]  worker_thread+0x4a/0x3d0
[ 2702.963412]  kthread+0xfd/0x130
[ 2702.963413]  ? process_one_work+0x3e0/0x3e0
[ 2702.963414]  ? kthread_park+0x90/0x90
[ 2702.963416]  ret_from_fork+0x22/0x40
[ 2702.963419] INFO: task kworker/2:1:226 blocked for more than 122 seconds.
[ 2702.963419]       Tainted: P        W  OEL    5.2.1-arch1-1-ARCH #1
[ 2702.963420] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2702.963420] kworker/2:1     D    0   226      2 0x80004000
[ 2702.963424] Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker [wireguard]
[ 2702.963425] Call Trace:
[ 2702.963427]  ? __schedule+0x27f/0x6d0
[ 2702.963429]  schedule+0x3d/0xc0
[ 2702.963431]  rwsem_down_read_failed+0x109/0x1b0
[ 2702.963434]  wg_noise_handshake_consume_initiation+0x38b/0x530 [wireguard]
[ 2702.963439]  wg_receive_handshake_packet+0x13d/0x4e0 [wireguard]
[ 2702.963441]  ? __switch_to_asm+0x40/0x70
[ 2702.963442]  ? __switch_to_asm+0x34/0x70
[ 2702.963446]  wg_packet_handshake_receive_worker+0x25/0x50 [wireguard]
[ 2702.963448]  process_one_work+0x1d1/0x3e0
[ 2702.963449]  worker_thread+0x4a/0x3d0
[ 2702.963451]  kthread+0xfd/0x130
[ 2702.963452]  ? process_one_work+0x3e0/0x3e0
[ 2702.963454]  ? kthread_park+0x90/0x90
[ 2702.963455]  ret_from_fork+0x22/0x40
[ 2702.963459] INFO: task kworker/6:1:246 blocked for more than 122 seconds.
[ 2702.963460]       Tainted: P        W  OEL    5.2.1-arch1-1-ARCH #1
[ 2702.963460] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2702.963461] kworker/6:1     D    0   246      2 0x80004000

Offline

#3 2019-07-22 08:52:33

nc99
Member
Registered: 2019-07-21
Posts: 3

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

Thanks for sharing. Yep, I mentioned in the original post that I have the same generation CPU (3900X). I'm thinking it might be related to the Ryzen 3000 CPUs. I remember there was that issue with systemd on Ryzen 3000 series due to the way it generated random numbers, and the dmesg mentions "get_random_u32" in both of our dmesg outputs.

Edit: If that's the case, my question is, is this an issue with Wireguard or is this something that should be fixed with a BIOS update?

Last edited by nc99 (2019-07-22 08:53:57)

Offline

#4 2019-07-22 09:08:07

progandy
Member
Registered: 2012-05-17
Posts: 5,263

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

Have you tried booting with the "nordrand" kernel parameter? And if wireguard runs out of entropy and that causes the stall, maybe try haveged or a hardware rng?

Last edited by progandy (2019-07-22 09:09:42)

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#5 2019-07-22 09:44:13

nc99
Member
Registered: 2019-07-21
Posts: 3

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

Thanks, progandy! smile I added the nordrand kernel parameter and Wireguard has now been running without issue for over 15 minutes. I'll mark this as solved since I can now run Wireguard, but is this something that should be fixed somewhere (perhaps Wireguard's code?) or is nordrand an acceptable long term solution?

Offline

#6 2019-07-22 10:01:47

progandy
Member
Registered: 2012-05-17
Posts: 5,263

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

nordrand prevents the kernel from using RdRand which is buggy on ryzen 3000. Systemd isn't using the kernel API, so they had to roll their own patch.

Maybe there should be a kernel patch to disable it automatically for this CPU series.

Last edited by progandy (2019-07-22 10:05:39)

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#7 2019-07-22 10:44:05

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

AFAIK AMD has already released a firmware fix to all of their motherboard manufacturers, they should have it repaged for download any day now...
https://www.phoronix.com/scan.php?page= … x-Zen2-Fix

No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#8 2021-03-16 23:30:19

solarkraft
Member
Registered: 2021-03-16
Posts: 2

Re: [SOLVED] Wireguard causes system to hang (Ryzen 3000 series CPUs)

I have/had this issue with an i7-8750H (Dell XPS 15 9570). nordrand may be working for me.

Update: It does not smile

Last edited by solarkraft (2021-03-16 23:34:41)

Offline

Pages: 1

Board footer

Powered by FluxBB