Suddenly, no DNS resolution

quequotion · 2015-10-21 04:54:15

This happened very suddenly, in the middle of an upgrade. The upgrade failed when it inexplicably couldn't find servers to download packages from.

I know systemd was upgraded among several other packages, but I was not able to retain any log of the upgrade.

DNS does not resolve. I was using systemd-networkd with dnsmasq and dnscrypt-proxy. I have thoroughly tested both dnsmasq and dnscrypt-proxy's configurations, which had not been changed. Both services start and run without error. I also tried disabling both and reconfiguring for systemd-networkd with systemd-resolved, but still DNS would not resolve.

EDIT: I haven't tried a different dnscrypt server yet, but I very much doubt DNS requests are being processed by dnscrypt-proxy.service at all.

I can ping IP addresses directly, but pings to domain names fail immediately. There's no delay--not even a fraction of a second, as if DNS resolution was intentionally aborted or never attempted.

I need a means to find out:
-Where are DNS request being sent to, if not the configured servers?
Only nslookup and drill produce results and only if no dns resolution daemon is available or systemd-resolved is available. Both consistently ignore dnsmasq, using only the DNS servers from my ISP.
-What is presently responsible for processing DNS, if not the configured daemon(s)?
-How could this happen, if there were no configuration changes and no related packages removed?

Nothing appears to be broken except that DNS will not resolve. I can't find anything related to DNS in the logs.

Last edited by quequotion (2015-10-23 02:56:49)

ayekat · 2015-10-21 06:49:42

Is systemd-resolved.service running? Or how do you configure resolv.conf?

Otherwise, what is the output of these commands?

ls -l /etc/resolv.conf
ls -l /run/systemd/resolve/resolv.conf

quequotion · 2015-10-21 08:37:27

ayekat wrote:

Is systemd-resolved.service running?

At the moment, yes. I have swapped back and forth many times, rebooting each time, between systemd-resolved and dnsmasq+dnscrypt-proxy.The two methods are not compatible, or at least cannot be reasonably configured for compatibility (edit: or can they?).

how do you configure resolv.conf?

I don't, not conciously. It pulls in two DNS servers from my router regardless of what is going with systemd services. I suspect this is DHCP auto-configuration.

what is the output of these commands?

ls -l /etc/resolv.conf

It exists and is not a symlink,

ls -l /run/systemd/resolve/resolv.conf

Also exists.

Both files specify the same nameservers, the ones specified on the router. Those servers work. They are the DNS servers for my service provider. Other devices connected to this router have no DNS problems.

EDIT: After a thorough scouring of the wiki, I've configured systemd-resolved to prepend dnsmasq's server, so the two methods can--in theory--work together.

However, despite correct configuration of systemd-resolved according to the wiki, correct configuration of dnsmasq according to the wiki, and correct configuration of dnscrypt-proxy according to the wiki, only a few specific programs are able to resolve DNS.

Those are nslookup and drill. Nothing else.

Last edited by quequotion (2015-10-23 02:59:26)

quequotion · 2015-10-22 05:38:15

Using the internet by the only means currently available on this machine: Tor Browser

How is this even possible?

I've been over and over and over all the possible configuration options... nothing appears to be out of place, yet DNS simply will not be resolved by anything on my system except nslookup and drill.

For example, epiphany cannot load any webpages and returns this:

Error resolving 'www.google.com': System error

Which is slightly different from the usual dns resolution failure:

Error resolving 'mail.google.com': Name or service not known

What the heck is going on here?

Last edited by quequotion (2015-10-22 06:20:26)

quequotion · 2015-10-22 06:42:23

Here are the systemd services I have runing:

systemctl --type=service
UNIT                               LOAD   ACTIVE SUB     DESCRIPTION
accounts-daemon.service            loaded active running Accounts Service
avahi-daemon.service               loaded active running Avahi mDNS/DNS-SD Stack
bluetooth.service                  loaded active running Bluetooth service
colord.service                     loaded active running Manage, Install and Generate Color Profiles
cups-browsed.service               loaded active running Make remote CUPS printers available locally
dbus.service                       loaded active running D-Bus System Message Bus
dnscrypt-proxy.service             loaded active running DNSCrypt client proxy
dnsmasq.service                    loaded active running A lightweight DHCP and caching DNS server
kmod-static-nodes.service          loaded active exited  Create list of required static device nodes for the current kernel
lightdm.service                    loaded active running Light Display Manager
lm_sensors.service                 loaded active exited  Initialize hardware monitoring sensors
openntpd.service                   loaded active running OpenNTP Daemon
org.cups.cupsd.service             loaded active running CUPS Scheduler
polkit.service                     loaded active running Authorization Manager
rtkit-daemon.service               loaded active running RealtimeKit Scheduling Policy Service
sshd.service                       loaded active running OpenSSH Daemon
systemd-binfmt.service             loaded active exited  Set Up Additional Binary Formats
systemd-journal-flush.service      loaded active exited  Flush Journal to Persistent Storage
systemd-journald.service           loaded active running Journal Service
systemd-logind.service             loaded active running Login Service
systemd-networkd.service           loaded active running Network Service
systemd-random-seed.service        loaded active exited  Load/Save Random Seed
systemd-remount-fs.service         loaded active exited  Remount Root and Kernel File Systems
systemd-resolved.service           loaded active running Network Name Resolution
systemd-sysctl.service             loaded active exited  Apply Kernel Variables
systemd-tmpfiles-setup-dev.service loaded active exited  Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service     loaded active exited  Create Volatile Files and Directories
systemd-udev-trigger.service       loaded active exited  udev Coldplug all Devices
systemd-udevd.service              loaded active running udev Kernel Device Manager
systemd-update-utmp.service        loaded active exited  Update UTMP about System Boot/Shutdown
systemd-user-sessions.service      loaded active exited  Permit User Sessions
systemd-vconsole-setup.service     loaded active exited  Setup Virtual Console
throttle-cut.service               loaded active exited  Throttle Cut
tor.service                        loaded active running Anonymizing Overlay Network
udisks2.service                    loaded active running Disk Manager
upower.service                     loaded active running Daemon for power management
user@1000.service                  loaded active running User Manager for UID 1000

quequotion · 2015-10-22 06:56:53

Alright, got some DNS resolution back!

I had to change /etc/nsswitch.conf and replace "resolve" in "hosts:" with "dns".

I have no memory of making any such change ("dns" appears to be the default) and have never had a problem like this before. If it's been "resolve" for however long it has been, then it would be nice to know why the behavior suddenly changed.

Unfortunately, it seems that dnsmasq and dnscrypt-proxy are still inexplicably ignored.

Last edited by quequotion (2015-10-22 10:35:08)

quequotion · 2015-10-22 07:19:13

So, on to debugging what couldn't possibly be is wrong with dnsmasq+dnscrypt-proxy....

/etc/dnsmasq.conf:

strict-order
no-resolv
no-poll
server=127.0.0.1#40
listen-address=127.0.0.1

The first three options are intended to make sure dns is resolved *only* through dnsmasq+dnscrypt-proxy, but this is overriden by systemd-resolved if there are other servers available in /run/systemd/resolve/resolv.conf (which I have now symlinked to /etc/resolv.conf, not that it makes any difference). The only affect they have on DNS resolution is to limit the available servers. In particular, I'd rather not have my local ISP's servers seeing/caching/logging my DNS requests, or any other unencrypted DNS servers.

Here's /usr/lib/systemd/system/dnscrypt-proxy.socket:

[Unit]
Description=dnscrypt-proxy listening socket

[Socket]
ListenStream=
ListenDatagram=
ListenStream=127.0.0.1:40
ListenDatagram=127.0.0.1:40

[Install]
WantedBy=sockets.target

Here's /usr/lib/systemd/system/dnscrypt-proxy.service:

[Unit]
Description=DNSCrypt client proxy
Requires=dnscrypt-proxy.socket

[Install]
Also=dnscrypt-proxy.socket
WantedBy=multi-user.target

[Service]
Type=simple
NonBlocking=true
ExecStart=/usr/bin/dnscrypt-proxy \
	  -r 128.199.248.105:27015 -N 2.dnscrypt-cert.d0wn.biz -k 9970:E22D:7F6C:967F:8AED:CEEB:FBC1:94B9:AF54:376E:2BF7:39F1:F466:CBC9:AFDB:2A62

At the moment, if I remove the ISP servers from the list in /etc/resolve.conf, leaving only 127.0.0.1 (dnsmasq), DNS resolution, for progams like ping, fails after a significant delay, as if it were actually trying. drill fails with:

Error: error sending query: Could not send or receive, because of network error

However, both services seem to be running without any trouble.

systemctl status dnsmasq
● dnsmasq.service - A lightweight DHCP and caching DNS server
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2015-10-22 16:01:32 JST; 25min ago
     Docs: man:dnsmasq(8)
  Process: 499 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS)
 Main PID: 1736 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           └─1736 /usr/bin/dnsmasq -k --enable-dbus --user=dnsmasq --pid-file

Oct 22 16:01:31 hostname systemd[1]: Starting A lightweight DHCP and caching DNS server...
Oct 22 16:01:32 hostname dnsmasq[499]: dnsmasq: syntax check OK.
Oct 22 16:01:32 hostname systemd[1]: Started A lightweight DHCP and caching DNS server.
Oct 22 16:01:32 hostname dnsmasq[1736]: started, version 2.75 cachesize 1500
Oct 22 16:01:32 hostname dnsmasq[1736]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Oct 22 16:01:32 hostname dnsmasq[1736]: DBus support enabled: connected to system bus
Oct 22 16:01:32 hostname dnsmasq[1736]: using nameserver 127.0.0.1#40
Oct 22 16:01:32 hostname dnsmasq[1736]: read /etc/hosts - 10711 addresses

systemctl status dnscrypt-proxy
● dnscrypt-proxy.service - DNSCrypt client proxy
   Loaded: loaded (/usr/lib/systemd/system/dnscrypt-proxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2015-10-22 16:02:22 JST; 25min ago
 Main PID: 15893 (dnscrypt-proxy)
   CGroup: /system.slice/dnscrypt-proxy.service
           └─15893 /usr/bin/dnscrypt-proxy -r 128.199.248.105:27015 -N 2.dnscrypt-cert.d0wn.biz -k 9970:E22D:7F6C:967F:8AED:CEEB:FBC1:94B9:AF54:376E:2BF7:39F1:F466:CBC9:AFDB:2A62

Oct 22 16:02:22 hostname systemd[1]: Started DNSCrypt client proxy.
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [NOTICE] Starting dnscrypt-proxy 1.6.0
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] Generating a new session key pair
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] Done
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] Server certificate #808464433 received
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] This certificate looks valid
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] Chosen certificate #808464433 is valid from [2015-02-12] to [2016-02-12]
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [INFO] Server key fingerprint is EB60:C492:AB1B:EEDF:4A1C:F774:6EE1:3051:84A8:5CD1:926E:69EA:3C88:CD95:3228:3F16
Oct 22 16:02:22 hostname dnscrypt-proxy[15893]: [NOTICE] Proxying from 127.0.0.1:40 to 128.199.248.105:27015

systemctl status dnscrypt-proxy.socket
● dnscrypt-proxy.socket - dnscrypt-proxy listening socket
   Loaded: loaded (/usr/lib/systemd/system/dnscrypt-proxy.socket; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2015-10-22 16:01:29 JST; 27min ago
   Listen: 127.0.0.1:40 (Stream)
           127.0.0.1:40 (Datagram)

Last edited by quequotion (2015-10-22 07:31:30)

quequotion · 2015-10-22 10:44:13

quequotion wrote:

only a few specific programs are able to resolve DNS.
Those are nslookup and drill. Nothing else.

Perhaps this is the distinction between programs that are compatible with systemd's "resolve" and those that rely on glibc's "dns".... not that any such problem should exist.

branch · 2015-10-22 17:04:27

The difference is that nslookup and drill are DNS tools so they only do DNS lookups. The glibc resolver looks to nsswitch.conf to determine what methods of name resolution to try.

A few points:

1) /etc/resolv.conf should contain a single entry of 127.0.0.1 since you are running a local nameserver. Neither /etc/resolv.conf nor /run/systemd/resolve/resolv.conf should be sym links; breaking any links here will prevent systemd-networkd from overwriting the contents of /etc/resolv.conf based on dhcp info. You may also want to set "resolvconf=NO" in /etc/resolvconf.conf to prevent resolvconf from changing /etc/resolv.conf.

2) Try using just dnscrypt without a caching nameserver first. Stop/disable systemd-resolved.service and dnsmasq.service, and configure dnscrypt-proxy.socket to listen on 127.0.0.1:53 so nss-dns and drill can find it (remember to daemon-reload and restart the .socket). You should have "hosts: files dns myhostname" in /etc/nsswitch.conf for this configuration.

3) Once you have the above working, then decide which of systemd-resolved or dnsmasq you want to use for a caching stub.

3a. systemd-resolved: start/enable systemd-resolved.service and change "dns" in the "hosts" line of /etc/nsswitch.conf to "resolve". Leave dnscrypt-proxy.socket on 127.0.0.1:53 for this configuration because nss-dns and drill need to be able to find it. The libc resolver will now use the cache provided by resolved, but drill will use dnscrypt directly.

3b. dnsmasq: Change dnscrypt-proxy.socket back to 127.0.0.1:40 so only dnsmasq finds it (again, remember to daemon-reload and restart the .socket). Then start/enable dnsmasq.service. Leave "dns" in the "hosts" line of /etc/nsswitch.conf for this configuration. Here, both the libc resolver and drill will use the cache provided by dnsmasq.

vyu223 · 2015-10-22 23:09:05

I've been having the same issues as OP. Although I have updated recently beforehand, I did not perform any updates or change any configs in /etc/ since the last boot before dns lookups stopped working. Since my last update (and any changes in /etc/, for that matter), I've performed multiple reboots and have been able to resolve dns through dnscrypt between each boot (with only 127.0.0.1 in /etc/resolv.conf, sticky bit enabled, /etc/resolvconf.conf configured to NOT try to edit /etc/resolv.conf). Which is why I'm completely baffled. Usually a breakage occurs for me after an update, but this one seems to have come out of nowhere.

I've checked my pacman log which verifies what I've said above. I also use etckeeper; I ran a `git log` on etc to check for any modifications, which also verifies what I've said above.

I should also note that I'm also using dnsmasq.

I have already tried using dnscrypt-proxy.socket (which invokes dnscrypt-proxy.service) by itself without dnsmasq as described in the post immediately above. However, I still have the same problem, so I'm getting the idea that the problem isn't with dnsmasq.

EDIT: I should also note that I'm in Sendai, Japan. I don't know if this is relevant, but I just noticed OP is also residing in Japan.

Last edited by vyu223 (2015-10-22 23:15:35)

branch · 2015-10-22 23:49:33

vyu223 wrote:

I should also note that I'm in Sendai, Japan. I don't know if this is relevant, but I just noticed OP is also residing in Japan.

Which remote dnscrypt resolver are you guys using? Have you tried a different one?

quequotion · 2015-10-23 01:46:33

branch wrote:

/etc/resolv.conf should contain a single entry of 127.0.0.1 since you are running a local nameserver.

Indeed, unfortunately my local nameserver (dnsmasq+dnscrypt) mysteriously does not respond to requests.

Neither /etc/resolv.conf nor /run/systemd/resolve/resolv.conf should be sym links;

Originally neither were. I changed this, after posting this thread, because it is the recommended configuration for systemd-resolved, and I realized it can theoretically integrate with dnsmasq+dnyscrypt.

breaking any links here will prevent systemd-networkd from overwriting the contents of /etc/resolv.conf based on dhcp info.

I never figured out what, exactly was overwriting resolv.conf. It would seem that systemd-resolved is in charge of it now.

You may also want to set "resolvconf=NO" in /etc/resolvconf.conf to prevent resolvconf from changing /etc/resolv.conf.

Never heard of that one! I'll do that right away.

Come to think of it, whose idea was it to have all these different daemons overwrite the same configuration file? It doesn't seem very "unix" to have programs overwriting a configuration file they don't own.

Try using just dnscrypt without a caching nameserver first.

I have already done this. Somehow, although the daemon appears to be working, it does not respond to requests.

Once you have the above working, then decide which of systemd-resolved or dnsmasq you want to use for a caching stub.

This was working for some time (there's another thread for when I set up dnsmasq+dnscrypt to get an approximate starting date), and apparently with the "resolve" stub.... then one day it just stopped....

dnsmasq: Change dnscrypt-proxy.socket back to 127.0.0.1:40 so only dnsmasq finds it (again, remember to daemon-reload and restart the .socket). Then start/enable dnsmasq.service. Leave "dns" in the "hosts" line of /etc/nsswitch.conf for this configuration. Here, both the libc resolver and drill will use the cache provided by dnsmasq.

This is basically the configuration I had at the begining. At the time, I had not enabled systemd-resolved, yet I found "resolve" in nsswitch.conf.....

vyu223 wrote:

EDIT: I should also note that I'm in Sendai, Japan. I don't know if this is relevant, but I just noticed OP is also residing in Japan

Yeah, it might time to try another server, and it wouldn't be the first time the Japanese dnscrypt servers were busted (apparently they get heavily DDOSed) but I'm not sure requests are even being sent out. I still suspect the problem has something to do with the software or configuration on our machines. Did you configure a local dnscrypt server in /usr/lib/systemd/system/dnscrypt-proxy.service?

Last edited by quequotion (2015-10-23 01:47:48)

VanillaFunk · 2015-10-24 20:24:18

I'm glad to see this is a thing and not just me alone. Has anyone found a solution to this as of yet? I see the resolver is working and forwarding but like OP dnsmasq / dnscrypt-proxy are ignored

neodreams · 2015-10-25 16:12:25

This is a major problem for me, i need to use encrypted DNS, will try to see what's happening with that.

VanillaFunk · 2015-10-27 22:17:51

as of updating today i seem to have my dns encryption working as desired. Can anyone else confirm this?

Arch Linux

#1 2015-10-21 04:54:15

Suddenly, no DNS resolution

#2 2015-10-21 06:49:42

Re: Suddenly, no DNS resolution

#3 2015-10-21 08:37:27

Re: Suddenly, no DNS resolution

#4 2015-10-22 05:38:15

Re: Suddenly, no DNS resolution

#5 2015-10-22 06:42:23

Re: Suddenly, no DNS resolution

#6 2015-10-22 06:56:53

Re: Suddenly, no DNS resolution

#7 2015-10-22 07:19:13

Re: Suddenly, no DNS resolution

#8 2015-10-22 10:44:13

Re: Suddenly, no DNS resolution

#9 2015-10-22 17:04:27

Re: Suddenly, no DNS resolution

#10 2015-10-22 23:09:05

Re: Suddenly, no DNS resolution

#11 2015-10-22 23:49:33

Re: Suddenly, no DNS resolution

#12 2015-10-23 01:46:33

Re: Suddenly, no DNS resolution

#13 2015-10-24 20:24:18

Re: Suddenly, no DNS resolution

#14 2015-10-25 16:12:25

Re: Suddenly, no DNS resolution

#15 2015-10-27 22:17:51

Re: Suddenly, no DNS resolution

Board footer