You are not logged in.
Docker + Pi-hole + systemd-resolved - DNS resolution don't work at host / Pi-hole container / other containers.
Scenario:
[ WAN ] - [ LAN [ DHCP/DNS [ Proxmox [ Arch [ Docker [ [ Pi-hole] [ container-x ] [ ... ] ] ] ] ] ]
Goal:
Arch VM as host use Pi-hole container as DNS server, so other containers.
Pi-hole container use LAN DNS servers as his upstream DNS servers.
Yesterday spent all day trying to solve this without success. Searched for solutions and tried every one possibility I found.
Then, installed another VM with Ubuntu 22.04 Server and used same configuration / steps and it worked straight!
I really would like help to find what I done wrong or need to change because I prefer use Arch.
To prevent conflict between DNS server on Pi-hole container and Arch host:
sudo sed --in-place "s/^#DNSStubListener=yes$/DNSStubListener=no/g" /etc/systemd/resolved.conf
I use Pi-hole and other container behind Caddy, so bellow docker-compose.yml of Pi-hole, Caddy and Uptime Kuma.
cat << 'EOF' | tee ~/docker/pi-hole/docker-compose.yml > /dev/null
version: "3"
services:
pi-hole:
hostname: pi.hole
container_name: pi-hole
image: pihole/pihole:latest
ports:
- "53:53/tcp"
- "53:53/udp"
- "67:67/udp"
#- "80:80/tcp"
environment:
TZ: 'America/Sao_Paulo'
WEBPASSWORD: 'Passw0rd!'
ADMIN_EMAIL: 'admin@example.com'
PIHOLE_DNS_: 10.4.20.7; 10.4.20.6; 10.4.21.6
DHCP_ACTIVE: false
WEBTHEME: default-dark
DNSMASQ_LISTENING: all
volumes:
- app_config:/etc/pihole
- dnsmasq_config:/etc/dnsmasq.d
restart: unless-stopped
volumes:
app_config:
dnsmasq_config:
networks:
default:
name: caddy_net
external: true
EOF
cat << 'EOF' | tee ~/docker/caddy/docker-compose.yml > /dev/null
version: "3.7"
services:
caddy:
image: caddy:2.5.0-alpine
hostname: caddy
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- $PWD/Caddyfile:/etc/caddy/Caddyfile
- $PWD/sites:/srv
- app_data:/data
- app_config:/config
volumes:
app_data:
app_config:
networks:
default:
name: caddy_net
external: true
EOF
cat << 'EOF' | tee ~/docker/uptime-kuma/docker-compose.yml > /dev/null
version: "3.3"
services:
uptime-kuma:
image: louislam/uptime-kuma:latest
container_name: uptime-kuma
volumes:
- app_data:/app/data
#ports:
# - 3001:3001
restart: unless-stopped
security_opt:
- no-new-privileges:true
volumes:
app_data:
networks:
default:
name: caddy_net
external: true
EOF
I don't know which other info provide now, so ask and I will provide info as soon as possible.
Thanks!
Last edited by peracchi (2022-04-29 20:12:38)
Offline
If I'm understanding you correctly - the DNS "chain" should be like this:
Arch/programs -> /etc/resolv.conf -> 127.0.0.1:53 -> systemd-resolved service -> Pi-Hole -> LAN DNS server
With this operation:
sudo sed --in-place "s/^#DNSStubListener=yes$/DNSStubListener=no/g" /etc/systemd/resolved.conf
you break the DNS chain after /etc/resolv.conf.
Last edited by -thc (2022-04-28 17:54:08)
Offline
The only reason to disable the DNS Stub Listener is to free port 53, that will be used by Pi-hole through Docker.
What I understood is that Pi-hole inside Docker substitute de DNS Listener.
Other thing I saw and still reading/learning about is that /etc/resolv.conf has little interference nowadays.
Some parameters are put into /etc/systemd/resolved.conf and they affect systemd-resolved.
Right now I am recreating the Arch VM to do all again and better document the process.
What I still do not figure out is why the same configuration on similar bare servers (Arch and Ubuntu) works on one and not on another.
I will try to compare and to understand the output of "resolvectl status" command on each server.
Offline
The only reason to disable the DNS Stub Listener is to free port 53, that will be used by Pi-hole through Docker.
If the docker container listens on 127.0.0.1:53, you do not need to disable the StubListener - it does that by itself (if the container is already running)
If the docker container doesn't listen on 127.0.0.1:53, the DNS chain is broken.
Last edited by -thc (2022-04-28 18:19:06)
Offline
I have checked that Docker (and Pi-hole) was listening on 127.0.0.1:53 with
clear ; netstat -tlpn | sort -t: -k2 -n ; echo
As said, reinstalling and soon will be able to execute commands and try suggestions.
Bad luck, now getting a problem with self-signed certificate that do not had yesterday.
Could not fetch an active mirror-list: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:997)>
Selecting mirror region require a least one region to be given as an option.
Could not fetch an active mirror-list: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:997)>
Selecting mirror region require a least one region to be given as an option.
^CTraceback (most recent call last):
File "/usr/bin/archinstall", line 33, in <module>
sys.exit(load_entry_point('archinstall==2.3.3', 'console_scripts', 'archinstall')())
File "/usr/lib/python3.10/site-packages/archinstall/__init__.py", line 198, in run_as_a_module
script.execute()
File "/usr/lib/python3.10/site-packages/archinstall/lib/profiles.py", line 180, in execute
self.spec.loader.exec_module(sys.modules[self.namespace])
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/lib/python3.10/site-packages/archinstall/examples/guided.py", line 382, in <module>
ask_user_questions()
File "/usr/lib/python3.10/site-packages/archinstall/examples/guided.py", line 76, in ask_user_questions
archinstall.arguments['mirror-region'] = archinstall.select_mirror_regions(archinstall.list_mirrors())
File "/usr/lib/python3.10/site-packages/archinstall/lib/mirrors.py", line 146, in list_mirrors
response = urllib.request.urlopen(url)
File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/lib/python3.10/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/lib/python3.10/ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1070, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
KeyboardInterrupt
And I have imported the certificate same way as yesterday, with:
root@archiso ~ # trust anchor proxy-ckp-chain.crt
Installing from "archlinux-2022.04.05-x86_64.iso" with
root@archiso ~ # archinstall
Offline
With DNS Stub Listener enabled, the output of
clear ; sudo netstat -tlpn | sort -t: -k2 -n ; echo
is
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::22 :::* LISTEN 283/sshd: /usr/bin/
tcp6 0 0 :::5355 :::* LISTEN 635/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 283/sshd: /usr/bin/
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 635/systemd-resolve
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN 635/systemd-resolve
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 635/systemd-resolve
And when I try to start Pi-hole, as expected,
cd ~/docker/pi-hole/ && docker-compose up -d
[+] Running 10/10
⠿ pi-hole Pulled 18.6s
⠿ c229119241af Pull complete 8.1s
⠿ f6250565bc1f Pull complete 12.4s
⠿ 4f4fb700ef54 Pull complete 12.5s
⠿ b2fbfafc8a42 Pull complete 12.6s
⠿ bcc0395f729f Pull complete 12.7s
⠿ 2c6ae85afe38 Pull complete 12.7s
⠿ f2019a383060 Pull complete 14.2s
⠿ 06779f18ab1d Pull complete 14.2s
⠿ f8cdbf9a46bd Pull complete 14.3s
[+] Running 2/3
⠿ Volume "pi-hole_app_config" Created 0.0s
⠿ Volume "pi-hole_dnsmasq_config" Created 0.0s
⠿ Container pi-hole Starting 0.5s
Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (8bee7991cc5927553071adb56544a096f15e30ce780d7cff57faf007d9eccd9a): Error starting userland proxy: listen tcp4 0.0.0.0:53: bind: address already in use
With DNS Stub Listener disabled,
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::22 :::* LISTEN 283/sshd: /usr/bin/
tcp6 0 0 :::5355 :::* LISTEN 720/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 283/sshd: /usr/bin/
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 720/systemd-resolve
and I can start Pi-hole, but
$ docker exec -it pi-hole /bin/bash
root@pi:/# cat /etc/resolv.conf
nameserver 127.0.0.11
options edns0 trust-ad ndots:0
root@pi:/# ping -c 1 google.com
ping: google.com: Temporary failure in name resolution
The challenge here is to discover what is different on Ubuntu Server 22.04 that even also using systemd-resolved I can disable DNS Stub Listener and same settings for Docker/Containers work and in Arch do not.
Offline
With DNS Stub Listener disabled,
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::22 :::* LISTEN 283/sshd: /usr/bin/ tcp6 0 0 :::5355 :::* LISTEN 720/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 283/sshd: /usr/bin/ tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 720/systemd-resolve
Most DNS requests will be served via UDP and netstat is deprecated.
Please post the output (with running pi-hole) of
ss -u -l -n -p | grep ":53 "
and
ss -t -l -n -p | grep ":53 "
Technically you shouldn't need systemd-resolved - only the file /etc/resolv.conf pointing to the pi-hole socket.
and I can start Pi-hole, but
$ docker exec -it pi-hole /bin/bash root@pi:/# cat /etc/resolv.conf nameserver 127.0.0.11 options edns0 trust-ad ndots:0 root@pi:/# ping -c 1 google.com ping: google.com: Temporary failure in name resolution
Why 127.0.0.11? Where does that come from?
Offline
Before run Pi-hole and disable DNS Stub Listener (needed because of port conflict)
$ sudo ss -u -l -n -p | grep ":53 "
UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=272,fd=19))
UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=272,fd=17))
$ sudo ss -t -l -n -p | grep ":53 "
LISTEN 0 4096 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=272,fd=20))
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=272,fd=18))
With DNS Stub Listener disabled
$ sudo ss -u -l -n -p | grep ":53 "
$ sudo ss -t -l -n -p | grep ":53 "
With Pi-hole running and DNS Stub Listener disabled
$ sudo ss -u -l -n -p | grep ":53 "
UNCONN 0 0 0.0.0.0:53 0.0.0.0:* users:(("docker-proxy",pid=864,fd=4))
UNCONN 0 0 [::]:53 [::]:* users:(("docker-proxy",pid=870,fd=4))
$ sudo ss -t -l -n -p | grep ":53 "
LISTEN 0 4096 0.0.0.0:53 0.0.0.0:* users:(("docker-proxy",pid=845,fd=4))
LISTEN 0 4096 [::]:53 [::]:* users:(("docker-proxy",pid=850,fd=4))
Why 127.0.0.11? Where does that come from?
Docker.
When I began to troubleshoot this I also thinked that this were the problem...
Some "solutions" on Google is to change "127.0.0.11" by "127.0.0.1".
Then I learned that this is the DNS server Docker runs to allow containers resolve their names by container name.
DNS services
By default, a container inherits the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.
5. On user-defined networks like alpine-net, containers can not only communicate by IP address, but can also resolve a container name to an IP address. This capability is called automatic service discovery. Let’s connect to alpine1 and test this out. alpine1 should be able to resolve alpine2 and alpine4 (and alpine1, itself) to IP addresses.
Inside containers, in my "arch-docker" VM, DNS works only between containers while in "jammy-docker", with same configs, DNS works normal, as expected.
ID 0a85d46acf43
NAME uptime-kuma
Image louislam/uptime-kuma:latest
PORTS 3001/tcp
COMMAND "/usr/bin/dumb-init …"
CREATED 2022-04-29 08:50:33 -0300 -03
STATUS Up 56 seconds (health: starting)
ID 286a7631e34e
NAME caddy
Image caddy:2.5.0-alpine
PORTS 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 2019/tcp
COMMAND "caddy run --config …"
CREATED 2022-04-29 08:49:59 -0300 -03
STATUS Up About a minute
ID f58d72c1cace
NAME pi-hole
Image pihole/pihole:latest
PORTS 0.0.0.0:53->53/udp, :::53->53/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:67->67/udp, :::53->53/tcp, :::67->67/udp, 80/tcp
COMMAND "/s6-init"
CREATED 2022-04-29 08:32:19 -0300 -03
STATUS Up 30 seconds (health: starting)
[admin@arch-docker ~]$ docker exec -it pi-hole /bin/bash
root@pi:/# ping -c 1 google.com
ping: google.com: Temporary failure in name resolution
root@pi:/# ping -c 1 caddy
PING caddy (172.18.0.3) 56(84) bytes of data.
64 bytes from caddy.caddy_net (172.18.0.3): icmp_seq=1 ttl=64 time=0.071 ms
--- caddy ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.071/0.071/0.071/0.000 ms
root@pi:/#
exit
[admin@arch-docker ~]$ docker exec -it uptime-kuma /bin/bash
root@0a85d46acf43:/app# ping -c 1 google.com
ping: google.com: Temporary failure in name resolution
root@0a85d46acf43:/app# ping -c 1 pi-hole
PING pi-hole (172.18.0.2) 56(84) bytes of data.
64 bytes from pi-hole.caddy_net (172.18.0.2): icmp_seq=1 ttl=64 time=0.062 ms
--- pi-hole ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms
root@0a85d46acf43:/app# exit
[admin@arch-docker ~]$
Important info I forgeted to say: versions of docker and docker-compose are the same in Arch and Ubuntu, respectivately Docker version 20.10.14, build a224086 and Docker Compose version 2.4.1.
References:
https://docs.docker.com/config/containe … s-services
https://docs.docker.com/network/network … e-networks
Last edited by peracchi (2022-04-29 12:52:06)
Offline
If the pi-hole is running and listening on port 53 - can you stop systemd-resolved, write a /etc/resolv.conf with just one line:
nameserver 127.0.0.1
and do name resolution on arch?
Offline
Tried, problem persist.
Containers cannot resolve DNS names besides their own names.
[admin@arch-docker ~]$ docker exec pi-hole ping -c1 hotmail.com
ping: hotmail.com: Temporary failure in name resolution
[admin@arch-docker ~]$ docker exec pi-hole ping -c1 caddy
PING caddy (172.18.0.3) 56(84) bytes of data.
64 bytes from caddy.caddy_net (172.18.0.3): icmp_seq=1 ttl=64 time=0.086 ms
--- caddy ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.086/0.086/0.086/0.000 ms
[admin@arch-docker ~]$ tail -n 3 /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .
[admin@arch-docker ~]$ sudo vi /etc/resolv.conf
[admin@arch-docker ~]$ tail -n 3 /etc/resolv.conf
nameserver 127.0.0.1
options edns0 trust-ad
search .
[admin@arch-docker ~]$ sudo systemctl stop systemd-resolved
[admin@arch-docker ~]$ docker exec pi-hole ping -c1 hotmail.com
ping: hotmail.com: Temporary failure in name resolution
[admin@arch-docker ~]$ docker exec pi-hole ping -c1 caddy
PING caddy (172.18.0.3) 56(84) bytes of data.
64 bytes from caddy.caddy_net (172.18.0.3): icmp_seq=1 ttl=64 time=0.046 ms
--- caddy ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.046/0.046/0.046/0.000 ms
[admin@arch-docker ~]$ ping -c1 hotmail.com
PING hotmail.com (204.79.197.212) 56(84) bytes of data.
64 bytes from a-0010.a-msedge.net (204.79.197.212): icmp_seq=1 ttl=114 time=19.7 ms
--- hotmail.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.739/19.739/19.739/0.000 ms
Offline
Restarted two VMs (Arch and Ubuntu).
DNS Stub Listener disabled on both (because Docker needs port 53 free for Pi-hole container).
Output of command
resolvctl status
differ between Arch and Ubuntu. Maybe the answer is here...
Arch
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
Current DNS Server: 127.0.0.1
DNS Servers: 127.0.0.1
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google
DNS Domain: ~.
Link 2 (ens18)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.4.20.7
DNS Servers: 10.4.20.7 10.4.20.6 10.4.21.6
Link 3 (br-0f53b7fd1f70)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 4 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 6 (vethca53ed6)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 8 (veth2ec9350)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 10 (vethb5a1638)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Ubuntu
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink
Link 2 (ens18)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 10.4.20.7 10.4.20.6 10.4.21.6
DNS Domain: mydomain.com
Link 3 (br-7628878b5e56)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 4 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 6 (veth6568b0f)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 8 (veth6a13484)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 10 (veth33a5c4a)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Offline
On ubuntu, systemd-resolved runs in "uplink" mode.
To mirror this in Arch, create a symlink /etc/resolv.conf pointing to /run/systemd/resolve/resolv.conf.
Offline
On ubuntu, systemd-resolved runs in "uplink" mode.
To mirror this in Arch, create a symlink /etc/resolv.conf pointing to /run/systemd/resolve/resolv.conf.
Thanks so much for the help, -thc!
Finally working!
cd /etc/ && sudo rm -f resolv.conf && sudo ln -s ../run/systemd/resolve/stub-resolv.conf resolv.conf && sudo systemctl restart systemd-resolved && cd -
Whoever gets here, don't forget you have to disable DNS Stub Listener to use Pi-hole in container as your DNS server.
sudo sed --in-place "s/^#DNSStubListener=yes$/DNSStubListener=no/g" /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved
Offline