You are not logged in.
Hi All,
I am just curious if anyone else is having issues with docker and linux 6.
Docker Version: Docker version 20.10.21, build baeda1f82a
Docker will work fine with linux 5.19.9-arch1-1 but it will not start correctly with any of the linux 6 kernels i have tried to include:
linux-6.0.2.arch1-1
linux-6.0.5.arch1-1
linux-mainline-6.1rc2-1
With any of the above linux 6 kernels, any docker command will just hang. Even trying to stop or restart the docker.service will never complete.
The logs stop at:
level=info msg="Loading containers: done."Under Linux 5.19.9, it works and the logs seem to continue with:
level=info msg="Loading containers: done."
level=info msg="Docker daemon" commit=3056208812 graphdriver(s)=btrfs version=20.10.21
level=info msg="Daemon has completed initialization"
systemd[1]: Started Docker Application Container Engine.
level=info msg="API listen on /run/docker.sock"Looking at the differences of the logs suggests to me this may have something to do with the graphdriver=btrfs on linux 6, but thats just a guess. Any help or insights are appreciated.
Thanks!
Last edited by djmwj1 (2022-11-01 10:16:28)
Offline
I don't have this figured out yet, but I think it may have something to do with the network stack.
The docker service seems to launch three processes:
├─6025 /usr/bin/dockerd -H fd://
├─6043 containerd --config /var/run/docker/containerd/containerd.toml --log-level debug
└─6095 modprobe -va bridge br_netfilterThe last process, modprobe -va bridge br_netfilter seems to be where it gets hung up.
Trying to issue the command on my own yields:
sudo modprobe -v br_netfilter
insmod /lib/modules/6.1.0-rc2-1-mainline/kernel/net/bridge/br_netfilter.ko.zst And then it just hangs.
I can't quite correlate it yet, but I do see a bug in the dmesg logs:
[ 125.372952] nct6775: Found NCT6798D or compatible chip at 0x2e:0x290
[ 125.582605] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 125.599979] BUG: kernel NULL pointer dereference, address: 00000000000001c8
[ 125.600389] #PF: supervisor write access in kernel mode
[ 125.600658] #PF: error_code(0x0002) - not-present page
[ 125.600920] PGD 0 P4D 0
[ 125.601172] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 125.601421] CPU: 21 PID: 3492 Comm: rasdaemon Tainted: G OE 6.1.0-rc2-1-mainline #1 5554208761d242ecc1007b5fc9e857b8adf804cd
[ 125.601683] Hardware name: ASUS System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 1502 07/13/2021
[ 125.601946] RIP: 0010:ring_buffer_wake_waiters+0x28/0xb0
[ 125.602217] Code: e9 90 66 0f 1f 00 41 54 31 d2 4c 8d 67 10 55 48 89 fd 53 83 fe ff 74 3b 48 8b 47 60 48 63 f6 48 8b 1c f0 48 81 c3 78 01 00 00 <48> 83 43 50 01 31 c9 31 d2 48 8d 7b 20 be 03 00 00 00 e8 21 ab f4
[ 125.602503] RSP: 0018:ffffb547022afdc0 EFLAGS: 00010206
[ 125.602793] RAX: ffff8aa7828b4800 RBX: 0000000000000178 RCX: 0000000000000015
[ 125.603085] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff8aa79a2dfd00
[ 125.603380] RBP: ffff8aa79a2dfd00 R08: ffff8aa9e0c7a660 R09: ffff8aa8d98df470
[ 125.603674] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8aa79a2dfd10
[ 125.603971] R13: ffff8aa79dbac8e0 R14: ffff8aa8d8bf5380 R15: 0000000000000000
[ 125.604267] FS: 00007fb5254ef740(0000) GS:ffff8ac66d540000(0000) knlGS:0000000000000000
[ 125.604568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 125.604868] CR2: 00000000000001c8 CR3: 000000011b0ac000 CR4: 0000000000350ee0
[ 125.605173] Call Trace:
[ 125.605477] <TASK>
[ 125.605780] tracing_buffers_release+0x4d/0x90
[ 125.606085] __fput+0x89/0x250
[ 125.606389] task_work_run+0x5d/0x90
[ 125.606692] exit_to_user_mode_prepare+0x1a7/0x1d0
[ 125.606996] syscall_exit_to_user_mode+0x1b/0x40
[ 125.607300] do_syscall_64+0x6b/0x90
[ 125.607603] ? exit_to_user_mode_prepare+0x145/0x1d0
[ 125.607905] ? syscall_exit_to_user_mode+0x1b/0x40
[ 125.608206] ? do_syscall_64+0x6b/0x90
[ 125.608506] ? do_syscall_64+0x6b/0x90
[ 125.608802] ? do_syscall_64+0x6b/0x90
[ 125.609096] ? do_syscall_64+0x6b/0x90
[ 125.609387] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 125.609680] RIP: 0033:0x7fb5256d1774
[ 125.609994] Code: eb b2 67 e8 1e 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 80 3d ad 8d 0e 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 44 c3 0f 1f 00 48 83 ec 18 89 7c 24 0c e8 d3
[ 125.610311] RSP: 002b:00007fff15a387d8 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 125.610630] RAX: 0000000000000000 RBX: 0000000000000040 RCX: 00007fb5256d1774
[ 125.610947] RDX: 0000000000000081 RSI: 00007fff15a38ee0 RDI: 0000000000000045
[ 125.611264] RBP: 00007fff15a3a3b0 R08: 00005569293e65d0 R09: 0000000000000000
[ 125.611583] R10: 0000000000000008 R11: 0000000000000202 R12: 00007fff15a389e0
[ 125.611897] R13: 0000000000000001 R14: 0000000000000081 R15: 00005569293e2290
[ 125.612213] </TASK>
[ 125.612526] Modules linked in: nfnetlink bridge nct6775 nct6775_core hwmon_vid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ipvlan 8021q garp mrp stp llc vfat fat intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi iwlmvm edac_mce_amd btusb snd_hda_intel btrtl mac80211 snd_intel_dspcfg btbcm eeepc_wmi snd_intel_sdw_acpi btintel snd_usb_audio libarc4 asus_wmi kvm_amd iwlwifi snd_usbmidi_lib btmtk ledtrig_audio snd_hda_codec kvm sparse_keymap snd_rawmidi snd_seq_device rapl platform_profile wmi_bmof pcspkr mousedev joydev bluetooth snd_hda_core usblp cfg80211 mc snd_hwdep ecdh_generic atlantic igb crc16 snd_pcm rfkill dca snd_timer macsec snd sp5100_tco mxm_wmi soundcore k10temp i2c_piix4 tpm_crb tpm_tis tpm_tis_core mac_hid acpi_cpufreq nfsd nfs_acl lockd i2c_dev auth_rpcgss dm_multipath grace sg crypto_user fuse sunrpc bpf_preload ip_tables x_tables usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm dm_mod btrfs blake2b_generic libcrc32c crc32c_generic xor
[ 125.612573] raid6_pq crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd nvme cryptd nvme_core ccp xhci_pci rng_core nvme_common xhci_pci_renesas vfio_pci vfio_pci_core irqbypass vfio_virqfd vfio_iommu_type1 vfio amdgpu drm_ttm_helper ttm video wmi gpu_sched drm_buddy drm_display_helper cec vendor_reset(OE) kvmfr(OE)
[ 125.615209] CR2: 00000000000001c8
[ 125.615618] ---[ end trace 0000000000000000 ]---
[ 125.616028] RIP: 0010:ring_buffer_wake_waiters+0x28/0xb0
[ 125.616440] Code: e9 90 66 0f 1f 00 41 54 31 d2 4c 8d 67 10 55 48 89 fd 53 83 fe ff 74 3b 48 8b 47 60 48 63 f6 48 8b 1c f0 48 81 c3 78 01 00 00 <48> 83 43 50 01 31 c9 31 d2 48 8d 7b 20 be 03 00 00 00 e8 21 ab f4
[ 125.616871] RSP: 0018:ffffb547022afdc0 EFLAGS: 00010206
[ 125.617305] RAX: ffff8aa7828b4800 RBX: 0000000000000178 RCX: 0000000000000015
[ 125.617741] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff8aa79a2dfd00
[ 125.618178] RBP: ffff8aa79a2dfd00 R08: ffff8aa9e0c7a660 R09: ffff8aa8d98df470
[ 125.618617] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8aa79a2dfd10
[ 125.619059] R13: ffff8aa79dbac8e0 R14: ffff8aa8d8bf5380 R15: 0000000000000000
[ 125.619502] FS: 00007fb5254ef740(0000) GS:ffff8ac66d540000(0000) knlGS:0000000000000000
[ 125.619948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 125.620393] CR2: 00000000000001c8 CR3: 000000011b0ac000 CR4: 0000000000350ee0
[ 127.721618] rfkill: input handler disabled
[ 127.806678] [drm] DSC precompute is not needed.
[ 157.829245] kauditd_printk_skb: 61 callbacks suppressedIf anyone can help steer me further, or has insights, I appreciate it.
Offline
Maybe this:
[ 125.582605] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Offline
Maybe this:
[ 125.582605] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
My understanding is this is normal whenever bridge is called as it is being deprecated for br_netfilter.
Offline
Ok so the plot thickens,
the kernel module br_netfilter indeed does not load.
sudo lsmod | grep br_netfilter
libkmod: kmod_module_get_holders: could not open '/sys/module/ib_core/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/br_netfilter/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/snd_seq/holders': No such file or directory
libkmod: kmod_module_get_holders: could not open '/sys/module/nf_tables/holders': No such file or directoryit seems that none of the other modules it relies on loads as well as seen above. take nf_tables for example:
sudo insmod nf_tables
insmod: ERROR: could not load module nf_tables: No such file or directoryIndeed, i confirmed manually that these modules do not exist in /sys/modules/
However, if i run modinfo it finds the modules somewhere else ( /lib/modules/6.0.6-arch1-1/kernel/net/netfilter/) , not sure if this is normal or expected:
modinfo nf_tables
filename: /lib/modules/6.0.6-arch1-1/kernel/net/netfilter/nf_tables.ko.zst
alias: nfnetlink-subsys-10
author: Patrick McHardy <kaber@trash.net>
license: GPL
srcversion: ADC3B93BA3C8F4FA773EDC7
depends: nfnetlink,libcrc32c
retpoline: Y
intree: Y
name: nf_tables
vermagic: 6.0.6-arch1-1 SMP preempt mod_unload
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: 32:D7:53:B9:02:DA:96:69:D2:E8:BA:B9:71:59:C0:F0:07:C9:16:38
sig_hashalgo: sha512
signature: 30:64:02:30:7C:37:C2:2D:98:75:ED:9E:5E:D1:34:66:2F:36:0E:F6:
6D:AF:76:1B:4D:86:B7:E1:48:7A:F5:E4:68:F4:07:70:11:96:B2:A5:
61:79:A5:19:E4:AA:24:30:05:EC:65:52:02:30:5E:D5:F7:71:94:7C:
59:F6:81:99:0C:76:27:CC:32:AB:FE:7E:6D:56:DC:0B:A9:3A:84:A0:
1C:CD:38:4D:F0:E4:1C:71:91:ED:F3:69:CB:EC:84:0F:6F:2C:44:52:
1E:EDSo it seems for whatever reason, these builtin modules are simply not being place in /sys/module.
Not entirely sure how to fix this issue, but i think this is probably getting close to the root of it.
Full output of lsmod:
Module Size Used by
libkmod: kmod_module_get_holders: could not open '/sys/module/ib_core/holders': No such file or directory
ib_core -2 -2
libkmod: kmod_module_get_holders: could not open '/sys/module/snd_seq/holders': No such file or directory
snd_seq -2 -2
libkmod: kmod_module_get_holders: could not open '/sys/module/nfnetlink/holders': No such file or directory
nfnetlink -2 -2
bridge 364544 0
nct6775 28672 0
nct6775_core 77824 1 nct6775
hwmon_vid 16384 1 nct6775
dm_thin_pool 98304 1
dm_persistent_data 118784 1 dm_thin_pool
dm_bio_prison 20480 1 dm_thin_pool
dm_bufio 45056 1 dm_persistent_data
ipvlan 40960 0
8021q 40960 0
garp 16384 1 8021q
mrp 20480 1 8021q
stp 16384 2 bridge,garp
llc 16384 3 bridge,stp,garp
intel_rapl_msr 20480 0
eeepc_wmi 16384 0
asus_wmi 69632 1 eeepc_wmi
ledtrig_audio 16384 1 asus_wmi
sparse_keymap 16384 1 asus_wmi
platform_profile 16384 1 asus_wmi
vfat 24576 1
wmi_bmof 16384 0
mxm_wmi 16384 0
fat 98304 1 vfat
iwlmvm 532480 0
intel_rapl_common 32768 1 intel_rapl_msr
snd_hda_codec_hdmi 86016 1
mac80211 1314816 1 iwlmvm
btusb 65536 0
snd_usb_audio 397312 6
amdgpu 10727424 19
snd_hda_intel 61440 1
btrtl 28672 1 btusb
libarc4 16384 1 mac80211
snd_usbmidi_lib 45056 1 snd_usb_audio
edac_mce_amd 57344 0
snd_intel_dspcfg 36864 1 snd_hda_intel
btbcm 24576 1 btusb
snd_intel_sdw_acpi 20480 1 snd_intel_dspcfg
gpu_sched 49152 1 amdgpu
snd_hda_codec 188416 2 snd_hda_codec_hdmi,snd_hda_intel
btintel 45056 1 btusb
snd_rawmidi 49152 1 snd_usbmidi_lib
iwlwifi 491520 1 iwlmvm
drm_buddy 20480 1 amdgpu
kvm_amd 172032 0
snd_hda_core 118784 3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
btmtk 16384 1 btusb
snd_seq_device 16384 2 snd_rawmidi
video 61440 2 asus_wmi,amdgpu
kvm 1146880 1 kvm_amd
mc 77824 1 snd_usb_audio
snd_hwdep 16384 2 snd_usb_audio,snd_hda_codec
mousedev 24576 0
rapl 16384 0
bluetooth 937984 6 btrtl,btmtk,btintel,btbcm,btusb
joydev 28672 0
pcspkr 16384 0
usblp 32768 0
drm_ttm_helper 16384 1 amdgpu
cfg80211 1126400 3 iwlmvm,iwlwifi,mac80211
snd_pcm 172032 5 snd_hda_codec_hdmi,snd_hda_intel,snd_usb_audio,snd_hda_codec,snd_hda_core
ecdh_generic 16384 1 bluetooth
ttm 94208 2 amdgpu,drm_ttm_helper
igb 294912 0
atlantic 245760 0
crc16 16384 1 bluetooth
snd_timer 49152 2 snd_pcm
drm_display_helper 212992 1 amdgpu
dca 16384 1 igb
rfkill 32768 7 iwlmvm,asus_wmi,bluetooth,cfg80211
sp5100_tco 20480 0
macsec 61440 1 atlantic
snd 131072 27 snd_seq_device,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_usb_audio,snd_usbmidi_lib,snd_hda_codec,snd_timer,snd_pcm,snd_rawmidi
cec 81920 1 drm_display_helper
soundcore 16384 1 snd
i2c_piix4 36864 0
k10temp 16384 0
wmi 45056 5 video,nct6775,asus_wmi,wmi_bmof,mxm_wmi
tpm_crb 20480 0
tpm_tis 16384 0
tpm_tis_core 36864 1 tpm_tis
mac_hid 16384 0
acpi_cpufreq 32768 0
nfsd 761856 5
nfs_acl 16384 1 nfsd
lockd 155648 1 nfsd
auth_rpcgss 159744 1 nfsd
grace 16384 2 nfsd,lockd
i2c_dev 24576 0
dm_multipath 45056 0
sg 49152 0
crypto_user 24576 0
sunrpc 708608 14 nfsd,auth_rpcgss,lockd,nfs_acl
fuse 176128 5
bpf_preload 24576 0
ip_tables 36864 0
x_tables 57344 1 ip_tables
usbhid 73728 0
dm_crypt 61440 7
cbc 16384 0
encrypted_keys 28672 1 dm_crypt
trusted 53248 2 encrypted_keys,dm_crypt
asn1_encoder 16384 1 trusted
tee 36864 1 trusted
tpm 102400 4 tpm_tis,trusted,tpm_crb,tpm_tis_core
dm_mod 192512 31 dm_crypt,dm_thin_pool,dm_multipath,dm_bufio
btrfs 1925120 3
blake2b_generic 20480 0
libcrc32c 16384 2 dm_persistent_data,btrfs
crc32c_generic 16384 0
xor 24576 1 btrfs
raid6_pq 122880 1 btrfs
crct10dif_pclmul 16384 1
crc32_pclmul 16384 0
crc32c_intel 24576 4
polyval_clmulni 16384 0
polyval_generic 16384 1 polyval_clmulni
gf128mul 16384 1 polyval_generic
ghash_clmulni_intel 16384 0
sha512_ssse3 53248 0
aesni_intel 393216 14
crypto_simd 16384 1 aesni_intel
cryptd 24576 9 crypto_simd,ghash_clmulni_intel
nvme 61440 4
nvme_core 208896 5 nvme
ccp 135168 1 kvm_amd
xhci_pci 20480 0
nvme_common 24576 1 nvme_core
rng_core 20480 2 ccp,tpm
xhci_pci_renesas 24576 1 xhci_pci
vfio_pci 16384 0
vfio_pci_core 81920 1 vfio_pci
irqbypass 16384 2 vfio_pci_core,kvm
vfio_virqfd 16384 1 vfio_pci_core
vfio_iommu_type1 45056 0
vfio 49152 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
vendor_reset 155648 0
kvmfr 20480 0Last edited by djmwj1 (2022-10-31 14:41:11)
Offline
[ 125.601421] CPU: 21 PID: 3492 Comm: rasdaemon Tainted: G OE 6.1.0-rc2-1-mainline #1 5554208761d242ecc1007b5fc9e857b8adf804cd
[ 125.601683] Hardware name: ASUS System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 1502 07/13/2021
[ 125.601946] RIP: 0010:ring_buffer_wake_waiters+0x28/0xb0https://bugs.archlinux.org/task/76354 although it should not be reproducible on 6.0.2
Offline
[ 125.601421] CPU: 21 PID: 3492 Comm: rasdaemon Tainted: G OE 6.1.0-rc2-1-mainline #1 5554208761d242ecc1007b5fc9e857b8adf804cd [ 125.601683] Hardware name: ASUS System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 1502 07/13/2021 [ 125.601946] RIP: 0010:ring_buffer_wake_waiters+0x28/0xb0https://bugs.archlinux.org/task/76354 although it should not be reproducible on 6.0.2
thanks loqs! It seems that disabling the rasdaemon solved this. I am not sure why the kernel error generated by rasdaemon caused the kernel to not load builtin modules right but that is apparently what happened. Thanks!!
Offline