You are not logged in.
Hi,
I have two Arch servers on my LAN that run a few containers via docker, but some time in the fairly recent past they've stopped being reachable via their port forward and in troubleshooting I've found they can't reach other hosts on the lan or the internet. It looks related to nftables or nat forwarding, but I haven't managed to figure it out. I'm going to dump information here and hope someone can give pointers. I will admit, I've never been great w/ iptables and worse at nftables.
The two hosts are apollo w/ IP 10.0.1.2 and hex w/ ip 10.0.1.22.
The two hosts can reach each other:
fryfrog@apollo ~ ❯ ping -c 2 10.0.1.22
PING 10.0.1.22 (10.0.1.22) 56(84) bytes of data.
64 bytes from 10.0.1.22: icmp_seq=1 ttl=64 time=0.251 ms
64 bytes from 10.0.1.22: icmp_seq=2 ttl=64 time=0.222 ms
--- 10.0.1.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1045ms
rtt min/avg/max/mdev = 0.222/0.236/0.251/0.014 msfryfrog@hex ~ ❯ ping -c 2 10.0.1.2
PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
64 bytes from 10.0.1.2: icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from 10.0.1.2: icmp_seq=2 ttl=64 time=0.305 ms
--- 10.0.1.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1011ms
rtt min/avg/max/mdev = 0.303/0.304/0.305/0.001 msHex can reach nginx on apollo running natively.
fryfrog@hex ~ ❯ curl --insecure https://10.0.1.2
Nothing here.The docker service and socket are stopped on both hosts.
fryfrog@apollo ~ ❯ sudo systemctl status docker.service docker.socket
○ docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Active: inactive (dead) since Tue 2026-04-07 08:28:34 PDT; 11s ago
Duration: 10h 12min 32.091s
Invocation: 81fc5221836e4c989a5ce9835a14a03c
TriggeredBy: ○ docker.socket
Docs: https://docs.docker.com
Process: 1195240 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 1195240 (code=exited, status=0/SUCCESS)
Mem peak: 41.5M
CPU: 8.902s
○ docker.socket - Docker Socket for the API
Loaded: loaded (/usr/lib/systemd/system/docker.socket; disabled; preset: disabled)
Active: inactive (dead) since Tue 2026-04-07 08:28:34 PDT; 11s ago
Duration: 10h 41min 36.301s
Invocation: 9c952707a391447faf1fd3ca9f72a8f4
Triggers: ● docker.service
Listen: /run/docker.sock (Stream)
Mem peak: 256K
CPU: 848usfryfrog@hex ~ ❯ sudo systemctl status docker.service docker.socket
○ docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Active: inactive (dead) since Tue 2026-04-07 08:28:31 PDT; 48s ago
Duration: 10h 52min 19.282s
Invocation: 4d391622cf6f409cbe2d4eaf01bfc23a
TriggeredBy: ○ docker.socket
Docs: https://docs.docker.com
Process: 1055745 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 1055745 (code=exited, status=0/SUCCESS)
Mem peak: 124M
CPU: 21.058s
Apr 07 08:28:21 hex systemd[1]: Stopping Docker Application Container Engine...
Apr 07 08:28:21 hex dockerd[1055745]: time="2026-04-07T08:28:21.266534328-07:00" level=info msg="Processing signal 'terminated'"
Apr 07 08:28:31 hex dockerd[1055745]: time="2026-04-07T08:28:31.274259022-07:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=438c74c41f53a732770540703c2c8d4c1f87f377d6b5d>
Apr 07 08:28:31 hex dockerd[1055745]: time="2026-04-07T08:28:31.293639851-07:00" level=info msg="received task-delete event from containerd" container=438c74c41f53a732770540703c2c8d4c1f87f377d6b5ddc85cd5cf5c09df55e2 modu>
Apr 07 08:28:31 hex dockerd[1055745]: time="2026-04-07T08:28:31.369170019-07:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Apr 07 08:28:31 hex dockerd[1055745]: time="2026-04-07T08:28:31.369509368-07:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Apr 07 08:28:31 hex dockerd[1055745]: time="2026-04-07T08:28:31.707238391-07:00" level=info msg="Daemon shutdown complete"
Apr 07 08:28:31 hex systemd[1]: docker.service: Deactivated successfully.
Apr 07 08:28:31 hex systemd[1]: Stopped Docker Application Container Engine.
Apr 07 08:28:31 hex systemd[1]: docker.service: Consumed 21.058s CPU time over 10h 52min 30.398s wall clock time, 124M memory peak.
○ docker.socket - Docker Socket for the API
Loaded: loaded (/usr/lib/systemd/system/docker.socket; disabled; preset: disabled)
Active: inactive (dead) since Tue 2026-04-07 08:28:31 PDT; 48s ago
Duration: 10h 57min 23.971s
Invocation: eb4eea2ac4c44c7d8334331252224431
Triggers: ● docker.service
Listen: /run/docker.sock (Stream)
Mem peak: 512K
CPU: 1ms
Apr 06 21:31:07 hex systemd[1]: Starting Docker Socket for the API...
Apr 06 21:31:07 hex systemd[1]: Listening on Docker Socket for the API.
Apr 07 08:28:31 hex systemd[1]: docker.socket: Deactivated successfully.
Apr 07 08:28:31 hex systemd[1]: Closed Docker Socket for the API.The iptables and nftables on both hosts are empty after flushing.
fryfrog@hex ~ ❯ sudo nft list ruleset
fryfrog@hex ~ ❯ sudo nft list tables
fryfrog@hex ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
fryfrog@hex ~ ❯ sudo iptables-legacy -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationfryfrog@apollo ~ ❯ sudo nft list ruleset
fryfrog@apollo ~ ❯ sudo nft list tables
fryfrog@apollo ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
fryfrog@apollo ~ ❯ sudo iptables-legacy -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationNow Docker is running and this is the iptables and ruleset.
fryfrog@apollo ~ ❯ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-04-07 08:34:31 PDT; 4s ago
Invocation: eaf226a1699644fb909ad3b35072b887
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1342043 (dockerd)
Tasks: 23
Memory: 38.3M (peak: 47.2M)
CPU: 570ms
CGroup: /system.slice/docker.service
└─1342043 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sockfryfrog@apollo ~ ❯ sudo nft list ruleset
table ip nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 165 bytes 8604 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 299 bytes 17940 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
table ip filter {
chain DOCKER {
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}fryfrog@apollo ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
target prot opt source destination
DOCKER-CT all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-INTERNAL all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-BRIDGE all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INTERNAL (1 references)
target prot opt source destination
Chain DOCKER-USER (1 references)
target prot opt source destinationfryfrog@hex ~ ❯ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-04-07 08:37:45 PDT; 3s ago
Invocation: 4a37bd4266984718acf6a1dc1347b110
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1139336 (dockerd)
Tasks: 15
Memory: 41.1M (peak: 49.3M)
CPU: 227ms
CGroup: /system.slice/docker.service
└─1139336 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.120262573-07:00" level=info msg="Deleting nftables IPv4 rules" error="exit status 1" output="Error: Could not process rule: No such file or directory\ndelet>
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.134261022-07:00" level=info msg="Deleting nftables IPv6 rules" error="exit status 1" output="Error: Could not process rule: No such file or directory\ndelet>
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.335875427-07:00" level=info msg="Loading containers: done."
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.346549686-07:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_F>
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.346606627-07:00" level=info msg="Docker daemon" commit=f78c987ad3 containerd-snapshotter=false storage-driver=overlay2 version=29.3.1
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.346633661-07:00" level=info msg="Initializing buildkit"
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.353363581-07:00" level=info msg="Completed buildkit initialization"
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.355074756-07:00" level=info msg="Daemon has completed initialization"
Apr 07 08:37:45 hex dockerd[1139336]: time="2026-04-07T08:37:45.355120792-07:00" level=info msg="API listen on /run/docker.sock"
Apr 07 08:37:45 hex systemd[1]: Started Docker Application Container Engine.fryfrog@hex ~ ❯ sudo nft list ruleset
table ip nat {
chain DOCKER {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 18 bytes 4117 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
table ip filter {
chain DOCKER {
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}fryfrog@hex ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
target prot opt source destination
DOCKER-CT all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-INTERNAL all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-BRIDGE all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INTERNAL (1 references)
target prot opt source destination
Chain DOCKER-USER (1 references)
target prot opt source destinationNow troubleshooting containers docker run --rm -e LISTENING_ADDRESS=":8888" -p 8888:8888 qmcgaw/port-checker and docker run -it nicolaka/netshoot are running on both hosts and this is nft and iptables output.
fryfrog@apollo ~ ❯ sudo nft list ruleset 08:45 [40/5857]
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain DOCKER {
iifname != "docker0" tcp dport 8888 counter packets 0 bytes 0 xt target "DNAT"
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 1927 bytes 142580 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 1529 bytes 91740 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER {
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 8888 counter packets 0 bytes 0 accept
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 172.17.0.3 iifname != "docker0" counter packets 0 bytes 0 drop
}
}fryfrog@apollo ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8888
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
target prot opt source destination
DOCKER-CT all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-INTERNAL all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-BRIDGE all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INTERNAL (1 references)
target prot opt source destination
Chain DOCKER-USER (1 references)
target prot opt source destinationfryfrog@apollo ~ ❯ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e4c668bbd960 nicolaka/netshoot "zsh" 3 minutes ago Up 3 minutes priceless_shockley
59f6d841c0f0 qmcgaw/port-checker "/port-checker" 4 minutes ago Up 4 minutes 8000/tcp, 0.0.0.0:8888->8888/tcp, [::]:8888->8888/tcp quirky_gauss
fryfrog@apollo ~ ❯ sudo docker port quirky_gauss
8888/tcp -> 0.0.0.0:8888
8888/tcp -> [::]:8888fryfrog@hex ~ ❯ sudo nft list ruleset 08:49 [40/150]
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain DOCKER {
iifname != "docker0" tcp dport 8888 counter packets 0 bytes 0 xt target "DNAT"
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 213 bytes 50254 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER {
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 8888 counter packets 0 bytes 0 accept
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 172.17.0.3 iifname != "docker0" counter packets 0 bytes 0 drop
}
}fryfrog@hex ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8888
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
target prot opt source destination
DOCKER-CT all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-INTERNAL all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-BRIDGE all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INTERNAL (1 references)
target prot opt source destination
Chain DOCKER-USER (1 references)
target prot opt source destinationfryfrog@hex ~ ❯ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8a65570bbb89 nicolaka/netshoot "zsh" About a minute ago Up About a minute agitated_wilbur
feb86adcd16a qmcgaw/port-checker "/port-checker" 2 minutes ago Up 2 minutes 8000/tcp, 0.0.0.0:8888->8888/tcp thirsty_archimedes
fryfrog@hex ~ ❯ sudo docker port thirsty_archimedes
8888/tcp -> 0.0.0.0:8888Forwarding is enabled overall and on the interface enp6s0f0 and enp2s0f0np0
fryfrog@apollo ~ ❯ sudo sysctl -a | grep forward
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.docker0.bc_forwarding = 0
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.docker0.mc_forwarding = 0
net.ipv4.conf.enp10s0.bc_forwarding = 0
net.ipv4.conf.enp10s0.forwarding = 1
net.ipv4.conf.enp10s0.mc_forwarding = 0
net.ipv4.conf.enp6s0f0.bc_forwarding = 0
net.ipv4.conf.enp6s0f0.forwarding = 1
net.ipv4.conf.enp6s0f0.mc_forwarding = 0
net.ipv4.conf.enp6s0f1.bc_forwarding = 0
net.ipv4.conf.enp6s0f1.forwarding = 1
net.ipv4.conf.enp6s0f1.mc_forwarding = 0
net.ipv4.conf.enp9s0.bc_forwarding = 0
net.ipv4.conf.enp9s0.forwarding = 1
net.ipv4.conf.enp9s0.mc_forwarding = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.veth36eb500.bc_forwarding = 0
net.ipv4.conf.veth36eb500.forwarding = 1
net.ipv4.conf.veth36eb500.mc_forwarding = 0
net.ipv4.conf.veth9cab507.bc_forwarding = 0
net.ipv4.conf.veth9cab507.forwarding = 1
net.ipv4.conf.veth9cab507.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.force_forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.force_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.docker0.force_forwarding = 0
net.ipv6.conf.docker0.forwarding = 0
net.ipv6.conf.docker0.mc_forwarding = 0
net.ipv6.conf.enp10s0.force_forwarding = 0
net.ipv6.conf.enp10s0.forwarding = 0
net.ipv6.conf.enp10s0.mc_forwarding = 0
net.ipv6.conf.enp6s0f0.force_forwarding = 0
net.ipv6.conf.enp6s0f0.forwarding = 0
net.ipv6.conf.enp6s0f0.mc_forwarding = 0
net.ipv6.conf.enp6s0f1.force_forwarding = 0
net.ipv6.conf.enp6s0f1.forwarding = 0
net.ipv6.conf.enp6s0f1.mc_forwarding = 0
net.ipv6.conf.enp9s0.force_forwarding = 0
net.ipv6.conf.enp9s0.forwarding = 0
net.ipv6.conf.enp9s0.mc_forwarding = 0
net.ipv6.conf.lo.force_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.veth36eb500.force_forwarding = 0
net.ipv6.conf.veth36eb500.forwarding = 0
net.ipv6.conf.veth36eb500.mc_forwarding = 0
net.ipv6.conf.veth9cab507.force_forwarding = 0
net.ipv6.conf.veth9cab507.forwarding = 0
net.ipv6.conf.veth9cab507.mc_forwarding = 0fryfrog@hex ~ ❯ sudo sysctl -a | grep forward
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.docker0.bc_forwarding = 0
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.docker0.mc_forwarding = 0
net.ipv4.conf.enp2s0f0np0.bc_forwarding = 0
net.ipv4.conf.enp2s0f0np0.forwarding = 1
net.ipv4.conf.enp2s0f0np0.mc_forwarding = 0
net.ipv4.conf.enp2s0f1np1.bc_forwarding = 0
net.ipv4.conf.enp2s0f1np1.forwarding = 1
net.ipv4.conf.enp2s0f1np1.mc_forwarding = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.veth4e9a0ca.bc_forwarding = 0
net.ipv4.conf.veth4e9a0ca.forwarding = 1
net.ipv4.conf.veth4e9a0ca.mc_forwarding = 0
net.ipv4.conf.vethc8c4ef4.bc_forwarding = 0
net.ipv4.conf.vethc8c4ef4.forwarding = 1
net.ipv4.conf.vethc8c4ef4.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.force_forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.force_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.docker0.force_forwarding = 0
net.ipv6.conf.docker0.forwarding = 0
net.ipv6.conf.docker0.mc_forwarding = 0
net.ipv6.conf.enp2s0f0np0.force_forwarding = 0
net.ipv6.conf.enp2s0f0np0.forwarding = 0
net.ipv6.conf.enp2s0f0np0.mc_forwarding = 0
net.ipv6.conf.enp2s0f1np1.force_forwarding = 0
net.ipv6.conf.enp2s0f1np1.forwarding = 0
net.ipv6.conf.enp2s0f1np1.mc_forwarding = 0
net.ipv6.conf.lo.force_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.veth4e9a0ca.force_forwarding = 0
net.ipv6.conf.veth4e9a0ca.forwarding = 0
net.ipv6.conf.veth4e9a0ca.mc_forwarding = 0
net.ipv6.conf.vethc8c4ef4.force_forwarding = 0
net.ipv6.conf.vethc8c4ef4.forwarding = 0
net.ipv6.conf.vethc8c4ef4.mc_forwarding = 0And now to try reaching the simple web server containers.
fryfrog@hex ~ ❯ curl http://127.0.0.1:8888
<html>
<head>
<title>Port checker page</title>
</head>
<body>
Listening address: :8888
<br>
Client address: 172.17.0.1:33662
<br>
Browser: Unknown 0
<br>
Device: Unknown
<br>
OS: Unknown 0
</body>
fryfrog@hex ~ ❯ curl http://10.0.1.22:8888
<html>
<head>
<title>Port checker page</title>
</head>
<body>
Listening address: :8888
<br>
Client address: 10.0.1.22:38982
<br>
Browser: Unknown 0
<br>
Device: Unknown
<br>
OS: Unknown 0
</body>
fryfrog@hex ~ ❯ curl --connect-timeout 10 http://10.0.1.2:8888
curl: (28) Connection timed out after 10002 millisecondsfryfrog@apollo ~ ❯ curl http://127.0.0.1:8888
<html>
<head>
<title>Port checker page</title>
</head>
<body>
Listening address: :8888
<br>
Client address: 172.17.0.1:60518
<br>
Browser: Unknown 0
<br>
Device: Unknown
<br>
OS: Unknown 0
</body>
fryfrog@apollo ~ ❯ curl http://10.0.1.2:8888
<html>
<head>
<title>Port checker page</title>
</head>
<body>
Listening address: :8888
<br>
Client address: 10.0.1.2:58216
<br>
Browser: Unknown 0
<br>
Device: Unknown
<br>
OS: Unknown 0
</body>
fryfrog@apollo ~ ❯ curl --connect-timeout 5 http://10.0.1.22:8888
curl: (28) Connection timed out after 5002 millisecondsAnd finally, trying to ping some IPs from the netshoot containers.
apollo ~ ping -c 2 10.0.1.2
PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
64 bytes from 10.0.1.2: icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from 10.0.1.2: icmp_seq=2 ttl=64 time=0.038 ms
--- 10.0.1.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1034ms
rtt min/avg/max/mdev = 0.038/0.055/0.073/0.017 ms
apollo ~ ping -c 2 10.0.1.1
PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
--- 10.0.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1029ms
apollo ~ ping -c 2 10.0.1.22
PING 10.0.1.22 (10.0.1.22) 56(84) bytes of data.
--- 10.0.1.22 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1057ms
apollo ~ ping -c 2 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1045ms hex ~ ping -c 2 10.0.1.22
PING 10.0.1.22 (10.0.1.22) 56(84) bytes of data.
64 bytes from 10.0.1.22: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 10.0.1.22: icmp_seq=2 ttl=64 time=0.066 ms
--- 10.0.1.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1053ms
rtt min/avg/max/mdev = 0.066/0.066/0.066/0.000 ms
hex ~ ping -c 2 10.0.1.2
PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
--- 10.0.1.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1033ms
hex ~ ping -c 2 10.0.1.1
PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
--- 10.0.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1012ms
hex ~ ping -c 2 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1014msAnd now, recent package updates to docker, iptables and nftables!
fryfrog@apollo ~ ❯ grep -E "(iptable|nftable|docker)" /var/log/pacman.log | tail -10
[2026-01-12T16:22:40-0800] [ALPM] upgraded docker (1:29.1.3-1 -> 1:29.1.4-1)
[2026-01-27T16:53:45-0800] [ALPM] upgraded docker (1:29.1.4-1 -> 1:29.2.0-1)
[2026-01-27T16:53:45-0800] [ALPM] upgraded docker-compose (5.0.1-1 -> 5.0.2-1)
[2026-02-05T15:46:55-0800] [ALPM] upgraded docker (1:29.2.0-1 -> 1:29.2.1-1)
[2026-03-02T10:35:26-0800] [ALPM] upgraded nftables (1:1.1.6-2 -> 1:1.1.6-3)
[2026-03-03T19:49:05-0800] [ALPM] upgraded docker-compose (5.0.2-1 -> 5.1.0-1)
[2026-03-08T13:48:31-0700] [ALPM] upgraded docker (1:29.2.1-1 -> 1:29.3.0-1)
[2026-03-20T06:57:47-0700] [ALPM] upgraded docker-compose (5.1.0-1 -> 5.1.1-1)
[2026-03-25T21:09:53-0700] [ALPM] upgraded docker (1:29.3.0-1 -> 1:29.3.1-1)
[2026-04-05T15:34:28-0700] [ALPM] upgraded iptables (1:1.8.11-2 -> 1:1.8.11-4)Last edited by fryfrog (2026-04-07 16:05:00)
Offline
And downgrading iptables to 1.8.11-2 fixes it. ![]()
Offline
I ran into issues with docker and networking this morning, and the fix for me was to enable the new nftables support for docker:
zsh ❯ cat /etc/docker/daemon.json
{
"firewall-backend": "nftables"
}...and then restarting the docker daemon. The upstream docs were useful for me too.
Offline
News on the front page and those "# Warning: iptables-legacy tables present, use iptables-legacy to see them" are important here. If you need the legacy interface, you need to switch packages.
Online
Thanks drewbug01 and Scimmia, I have no attachment to iptables so don't mind moving to nftables.
I've made the change to the docker daemon.json, stopped docker, flushed all the iptables and nftables and restarted docker. Now its *only* creating nft rules, but the hosts still can't reach containers on each other.
fryfrog@hex ~ ❯ sudo nft list ruleset 09:37 [53/1472]
table ip docker-bridges {
map filter-forward-in-jumps {
type ifname : verdict
elements = { "docker0" : jump filter-forward-in__docker0 }
}
map filter-forward-out-jumps {
type ifname : verdict
elements = { "docker0" : jump filter-forward-out__docker0 }
}
map nat-postrouting-in-jumps {
type ifname : verdict
elements = { "docker0" : jump nat-postrouting-in__docker0 }
}
map nat-postrouting-out-jumps {
type ifname : verdict
elements = { "docker0" : jump nat-postrouting-out__docker0 }
}
chain filter-FORWARD {
type filter hook forward priority filter; policy accept;
oifname vmap @filter-forward-in-jumps
iifname vmap @filter-forward-out-jumps
}
chain nat-OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 fib daddr type local counter packets 1 bytes 60 jump nat-prerouting-and-output
}
chain nat-POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
iifname vmap @nat-postrouting-out-jumps
oifname vmap @nat-postrouting-in-jumps
}
chain nat-PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
fib daddr type local counter packets 343 bytes 63405 jump nat-prerouting-and-output
}
chain nat-prerouting-and-output {
iifname != "docker0" tcp dport 8888 counter packets 8 bytes 480 dnat to 172.17.0.2:8888 comment "DNAT"
}
chain raw-PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop comment "DROP DIRECT ACCESS"
ip daddr 172.17.0.3 iifname != "docker0" counter packets 0 bytes 0 drop comment "DROP DIRECT ACCESS"
}
chain filter-forward-in__docker0 {
ct state established,related counter packets 0 bytes 0 accept
iifname "docker0" counter packets 0 bytes 0 accept comment "ICC"
ip daddr 172.17.0.2 tcp dport 8888 counter packets 7 bytes 420 accept
counter packets 0 bytes 0 drop comment "UNPUBLISHED PORT DROP"
}
chain filter-forward-out__docker0 {
ct state established,related counter packets 0 bytes 0 accept
counter packets 2 bytes 168 accept comment "OUTGOING"
}
chain nat-postrouting-in__docker0 {
}
chain nat-postrouting-out__docker0 {
oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 0 bytes 0 masquerade comment "MASQUERADE"
}
}
table ip6 docker-bridges {
map filter-forward-in-jumps {
type ifname : verdict
}
map filter-forward-out-jumps {
type ifname : verdict
}
map nat-postrouting-in-jumps {
type ifname : verdict
}
map nat-postrouting-out-jumps {
type ifname : verdict
}
chain filter-FORWARD {
type filter hook forward priority filter; policy accept;
oifname vmap @filter-forward-in-jumps
iifname vmap @filter-forward-out-jumps
}
chain nat-OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 fib daddr type local counter packets 0 bytes 0 jump nat-prerouting-and-output
}
chain nat-POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
iifname vmap @nat-postrouting-out-jumps
oifname vmap @nat-postrouting-in-jumps
}
chain nat-PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
fib daddr type local counter packets 0 bytes 0 jump nat-prerouting-and-output
}
chain nat-prerouting-and-output {
}
chain raw-PREROUTING {
type filter hook prerouting priority raw; policy accept;
}
}fryfrog@hex ~ ❯ sudo iptables -nL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
fryfrog@hex ~ ❯ sudo iptables-legacy -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationI don't have firewall setup on either system, but perhaps my default /etc/nftables.conf is too restrictive?
Offline
That legacy policy
Chain FORWARD (policy DROP)is probably the culprit.
Check all your self-defined (outside docker) nftables/iptables rulesets that are in place.
Offline
That legacy policy
Chain FORWARD (policy DROP)is probably the culprit.
Check all your self-defined (outside docker) nftables/iptables rulesets that are in place.
Dear lord, you were right! It was there the whole time, I just totally missed it. I set it to ACCEPT and now they can both reach each other like they should!
Thanks!
Offline