kernel panic possibly related to nfsv4

beholder · 2025-04-17 21:58:52

Hello, I've been having daily kernel panics on my Arch Linux machine and am wondering if anyone has seen this and know how to fix it?

This machine runs qbittorrent and downloads to an NFSv4 share.

Recently I've been running into kernel panics daily pretty much and the only way to get NFS share to respond again is to reboot the machine. It will work for a while, but then eventually the same kernel panic will occur again...

$ uname -a
Linux optiplex 6.14.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 18:43:59 +0000 x86_64 GNU/Linux

$ journalctl -b 0
...
Apr 17 06:00:00 optiplex systemd-helper[533730]: Running timeline for 'root'.
Apr 17 06:00:00 optiplex systemd[1]: snapper-timeline.service: Deactivated successfully.
Apr 17 06:01:00 optiplex systemd[1]: snapperd.service: Deactivated successfully.
Apr 17 06:09:27 optiplex kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
Apr 17 06:09:27 optiplex kernel: #PF: supervisor instruction fetch in kernel mode
Apr 17 06:09:27 optiplex kernel: #PF: error_code(0x0010) - not-present page
Apr 17 06:09:27 optiplex kernel: PGD 0 P4D 0 
Apr 17 06:09:27 optiplex kernel: Oops: Oops: 0010 [#1] PREEMPT SMP NOPTI
Apr 17 06:09:27 optiplex kernel: CPU: 21 UID: 0 PID: 537159 Comm: kworker/u96:12 Not tainted 6.14.2-arch1-1 #1 51440b8a0cc8bb91764dac94f6c2b53455e5a907
Apr 17 06:09:27 optiplex kernel: Hardware name: Dell Inc. OptiPlex 7000/0F1HC1, BIOS 1.4.1 07/20/2022
Apr 17 06:09:27 optiplex kernel: Workqueue: xprtiod xs_stream_data_receive_workfn [sunrpc]
Apr 17 06:09:27 optiplex kernel: RIP: 0010:0x0
Apr 17 06:09:27 optiplex kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Apr 17 06:09:27 optiplex kernel: RSP: 0018:ffffba9c4c6539f0 EFLAGS: 00010206
Apr 17 06:09:27 optiplex kernel: RAX: ffff979493c80000 RBX: 0000000000000000 RCX: 0000000000000000
Apr 17 06:09:27 optiplex kernel: RDX: 000000000000053c RSI: ffff979444c3c9aa RDI: ffff979493c8053c
Apr 17 06:09:27 optiplex kernel: RBP: ffffba9c4c653d68 R08: 0000000000000000 R09: ffffffffb7bfad70
Apr 17 06:09:27 optiplex kernel: R10: 000000000000006c R11: ffff979403d63000 R12: ffff979444c3c46e
Apr 17 06:09:27 optiplex kernel: R13: ffff979517663000 R14: 000000000000053c R15: 000000000000053c
Apr 17 06:09:27 optiplex kernel: FS:  0000000000000000(0000) GS:ffff97a34fa80000(0000) knlGS:0000000000000000
Apr 17 06:09:27 optiplex kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 17 06:09:27 optiplex kernel: CR2: ffffffffffffffd6 CR3: 0000000743c22006 CR4: 0000000000f72ef0
Apr 17 06:09:27 optiplex kernel: PKRU: 55555554
Apr 17 06:09:27 optiplex kernel: Call Trace:
Apr 17 06:09:27 optiplex kernel:  <TASK>
Apr 17 06:09:27 optiplex kernel:  ? _copy_to_iter+0x24a/0x5b0
Apr 17 06:09:27 optiplex kernel:  ? __skb_datagram_iter+0x7b/0x2e0
Apr 17 06:09:27 optiplex kernel:  ? __check_object_size+0x82/0x210
Apr 17 06:09:27 optiplex kernel:  ? __skb_datagram_iter+0x7b/0x2e0
Apr 17 06:09:27 optiplex kernel:  ? __pfx_simple_copy_to_iter+0x10/0x10
Apr 17 06:09:27 optiplex kernel:  ? skb_copy_datagram_iter+0x34/0x90
Apr 17 06:09:27 optiplex kernel:  ? tcp_recvmsg_locked+0x7af/0xa00
Apr 17 06:09:27 optiplex kernel:  ? tcp_recvmsg+0x87/0x200
Apr 17 06:09:27 optiplex kernel:  ? inet_recvmsg+0x54/0x130
Apr 17 06:09:27 optiplex kernel:  ? security_socket_recvmsg+0x5d/0x1a0
Apr 17 06:09:27 optiplex kernel:  ? sock_recvmsg+0xa6/0xd0
Apr 17 06:09:27 optiplex kernel:  ? xs_read_stream_request.constprop.0+0x2d7/0x570 [sunrpc 7006ce0128ebdc931ffa7b16e2e436a0bfc98294]
Apr 17 06:09:27 optiplex kernel:  ? xs_read_stream.constprop.0+0x389/0x520 [sunrpc 7006ce0128ebdc931ffa7b16e2e436a0bfc98294]
Apr 17 06:09:27 optiplex kernel:  ? xs_stream_data_receive_workfn+0x6c/0x130 [sunrpc 7006ce0128ebdc931ffa7b16e2e436a0bfc98294]
Apr 17 06:09:27 optiplex kernel:  ? process_one_work+0x17b/0x330
Apr 17 06:09:27 optiplex kernel:  ? worker_thread+0x2ce/0x3f0
Apr 17 06:09:27 optiplex kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 17 06:09:27 optiplex kernel:  ? kthread+0xec/0x230
Apr 17 06:09:27 optiplex kernel:  ? __pfx_kthread+0x10/0x10
Apr 17 06:09:27 optiplex kernel:  ? ret_from_fork+0x31/0x50
Apr 17 06:09:27 optiplex kernel:  ? __pfx_kthread+0x10/0x10
Apr 17 06:09:27 optiplex kernel:  ? ret_from_fork_asm+0x1a/0x30
Apr 17 06:09:27 optiplex kernel:  </TASK>
Apr 17 06:09:27 optiplex kernel: Modules linked in: ipt_REJECT nf_reject_ipv4 xt_connmark xt_mark iptable_mangle xt_comment nf_conntrack_netlink xt_nat xt_tcpudp veth rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio sunrpc netfs xt_conntrack xt_MASQUERADE bridge stp llc ip6table_nat ip6table_filter xt_set ip_set iptable_nat xt_addrtype iptable_filter xfrm_user xfrm_algo overlay nft_fib_ipv6 nft_masq nft_nat nft_fib_ipv4 nft_fib nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables vfat fat snd_ctl_led xe drm_gpuvm snd_hda_codec_realtek gpu_sched drm_exec snd_hda_codec_generic drm_suballoc_helper snd_hda_scodec_component drm_ttm_helper snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda_mlink snd_sof_intel_hda snd_hda_codec_hdmi snd_sof_pci snd_sof_xtensa_dsp intel_uncore_frequency intel_uncore_frequency_common snd_sof snd_sof_utils snd_soc_acpi_intel_match
Apr 17 06:09:27 optiplex kernel:  snd_soc_acpi_intel_sdca_quirks soundwire_generic_allocation snd_soc_acpi soundwire_bus wireguard snd_soc_sdca curve25519_x86_64 snd_soc_avs libchacha20poly1305 chacha_x86_64 snd_soc_hda_codec poly1305_x86_64 snd_hda_ext_core x86_pkg_temp_thermal libcurve25519_generic intel_powerclamp snd_soc_core iwlmvm libchacha ip6_udp_tunnel snd_compress coretemp i915 ac97_bus udp_tunnel snd_pcm_dmaengine mac80211 snd_hda_intel kvm_intel snd_intel_dspcfg processor_thermal_device_pci btusb snd_intel_sdw_acpi processor_thermal_device libarc4 snd_hda_codec drm_buddy btrtl processor_thermal_wt_hint kvm snd_hda_core iTCO_wdt processor_thermal_rfim btintel ttm intel_rapl_msr processor_thermal_rapl snd_hwdep iwlwifi i2c_algo_bit dell_pc spi_nor btbcm intel_pmc_bxt dell_wmi rapl snd_pcm intel_rapl_common mei_hdcp iTCO_vendor_support btmtk drm_display_helper mei_wdt mei_pxp ee1004 mtd intel_cstate dell_smbios cfg80211 bluetooth platform_profile intel_uncore snd_timer dcdbas processor_thermal_wt_req dell_wmi_sysman spi_intel_pci
Apr 17 06:09:27 optiplex kernel:  i2c_i801 cec dell_wmi_ddv dell_smm_hwmon mei_me intel_pmc_core processor_thermal_power_floor snd i2c_smbus firmware_attributes_class pcspkr dell_wmi_descriptor wmi_bmof spi_intel rfkill intel_gtt processor_thermal_mbox soundcore i2c_mux pmt_telemetry mei int3400_thermal int340x_thermal_zone pmt_class intel_hid acpi_thermal_rel acpi_pad intel_vsec sparse_keymap acpi_tad mac_hid loop nfnetlink zram 842_decompress 842_compress lz4hc_compress lz4_compress dm_crypt cbc encrypted_keys trusted asn1_encoder tee polyval_clmulni polyval_generic ghash_clmulni_intel ixgbe sha512_ssse3 hid_generic sha256_ssse3 mdio sha1_ssse3 mdio_devres e1000e nvme aesni_intel ucsi_acpi libphy intel_lpss_pci crypto_simd video usbhid typec_ucsi dca nvme_core intel_lpss cryptd roles ptp idma64 nvme_auth typec pps_core wmi pinctrl_alderlake dm_mod iptable_raw ip6_tables ip_tables x_tables
Apr 17 06:09:27 optiplex kernel: CR2: 0000000000000000
Apr 17 06:09:27 optiplex kernel: ---[ end trace 0000000000000000 ]---
Apr 17 06:09:27 optiplex kernel: RIP: 0010:0x0
Apr 17 06:09:27 optiplex kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Apr 17 06:09:27 optiplex kernel: RSP: 0018:ffffba9c4c6539f0 EFLAGS: 00010206
Apr 17 06:09:27 optiplex kernel: RAX: ffff979493c80000 RBX: 0000000000000000 RCX: 0000000000000000
Apr 17 06:09:27 optiplex kernel: RDX: 000000000000053c RSI: ffff979444c3c9aa RDI: ffff979493c8053c
Apr 17 06:09:27 optiplex kernel: RBP: ffffba9c4c653d68 R08: 0000000000000000 R09: ffffffffb7bfad70
Apr 17 06:09:27 optiplex kernel: R10: 000000000000006c R11: ffff979403d63000 R12: ffff979444c3c46e
Apr 17 06:09:27 optiplex kernel: R13: ffff979517663000 R14: 000000000000053c R15: 000000000000053c
Apr 17 06:09:27 optiplex kernel: FS:  0000000000000000(0000) GS:ffff97a34fa80000(0000) knlGS:0000000000000000
Apr 17 06:09:27 optiplex kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 17 06:09:27 optiplex kernel: CR2: ffffffffffffffd6 CR3: 0000000743c22006 CR4: 0000000000f72ef0
Apr 17 06:09:27 optiplex kernel: PKRU: 55555554
Apr 17 06:09:27 optiplex kernel: note: kworker/u96:12[537159] exited with irqs disabled
Apr 17 06:12:28 optiplex kernel: nfs: server 10.10.40.3 not responding, still trying
Apr 17 06:33:10 optiplex systemd[1]: Started Daily Cleanup of Snapper Snapshots.
Apr 17 06:33:10 optiplex systemd[1]: Starting DBus interface for snapper...
Apr 17 06:33:10 optiplex systemd[1]: Started DBus interface for snapper.
...

gromit · 2025-04-18 09:11:00

Is this a problem that only recently started to occur?

seth · 2025-04-18 14:12:11

OP wrote:

Recently I've been running into kernel panics daily pretty much

only way to get NFS share to respond again

Can you (force) umount the share and reload the modules?
mount options? ("mount" will print them all) - notably reg. the cache handling.

Apr 17 06:12:28 optiplex kernel: nfs: server 10.10.40.3 not responding, still trying

Can you otherwise still reach the (I'm guessing) NAS, resp. does the network work at all otherwise?
iwlwifi or e1000e connection?

Is the journal filtered or is that all that's happening during those 30 minutes?

Finally: does it affect the LTS kernel?

beholder · 2025-04-18 17:54:51

the problem started on April 12th. Back then I was running

- Linux version 6.13.2-arch1-1
- qbittorrent (docker - linuxserver/qbittorrent:4.6.7-libtorrentv1) -> I had libtorrent v1 because v2 was causing NFSv4 crashes some months ago. Also many months ago qbittorrent v5 had been released but had some major bugs initially so that's why I was on the previous version.

Yesterday I switched the docker image to the latest - linuxserver/qbittorrent:latest (qbit v5.0.4 and this image uses libtorrent v2) to see if it helps.

These were the dates it crashed this last week.

$ sudo journalctl --grep 'NULL pointer dereference'
-- Boot 8f1089b92d7d4f8b86131fa85b0f0e07 --
Apr 12 10:25:04 optiplex kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
-- Boot bcd4d1150373436b8f0835415376d078 --
Apr 16 13:25:36 optiplex kernel: BUG: kernel NULL pointer dereference, address: 00000000000004e8
-- Boot 3dad3f3e50d94447a31449cc8971b715 --
Apr 17 06:09:27 optiplex kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000

So far qbittorrent has been running for 12hrs and there has been no crashes so far, but I'll keep an eye on it.

Can you otherwise still reach the (I'm guessing) NAS, resp. does the network work at all otherwise?
iwlwifi or e1000e connection?
Is the journal filtered or is that all that's happening during those 30 minutes?
Finally: does it affect the LTS kernel?

When that crash happens, I can't reach the NFS server. Example if I do `ls -alh /mnt/share/` it never completes and stops responding.
What I do when that kernel crash happens is to reboot and then things start working again.
I haven't tried the LTS kernel.
I don't think that it's a new linux bug. I've been running Linux 6.13.2-arch1-1 since February 17th and never had those crashes until last week.
Weird thing is that the linuxserver/qbittorrent:4.6.7-libtorrentv1 image I was using hasn't been updated since 7 months ago (Sep 29, 2024).
I'm not sure what caused this crash to start happening...

Last edited by beholder (2025-04-18 18:14:49)

seth · 2025-04-18 18:40:38

I rather meant whether you can still "ping 10.10.40.3" or "ping 8.8.8.8" or at least "ping _gateway" when this happens.

problem started on April 12th. Back then I was running Linux version 6.13.2-arch1-1 … I've been running Linux 6.13.2-arch1-1 since February 17th and never had those crashes until last week

Did you change anything about the NAS or the network configuration at large (updates, new switch, new cables, …)?

However

I had libtorrent v1 because v2 was causing NFSv4 crashes some months ago

contradicts the other assertion, you very much /did/ have these problems in the past and it's maybe just a matter of how many seeds you find for the last season of white lo… how fast you can download linux distributions via bittorrent?

Arch Linux

#1 2025-04-17 21:58:52

kernel panic possibly related to nfsv4

#2 2025-04-18 09:11:00

Re: kernel panic possibly related to nfsv4

#3 2025-04-18 14:12:11

Re: kernel panic possibly related to nfsv4

#4 2025-04-18 17:54:51

Re: kernel panic possibly related to nfsv4

#5 2025-04-18 18:40:38

Re: kernel panic possibly related to nfsv4

Board footer