Arch Linux

TheChuckster

Member

Registered: 2013-07-19

Posts: 14

glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

dig has always worked as I expected, but glibc name resolution (and everything using it... ping, Firefox, etc.) started failing.

Two problems suddenly started occurring:

1.) NetworkManager kept pulling in a ::1 "IPv6 localhost" IP, I don't run a local DNS server on my own machine, nor do I use IPv6. NetworkManager edits /etc/resolv.conf on its own, but I also noticed that the system doesn't even pay attention to it anymore.

I managed to prevent NetworkManager from overwriting this file, and threw in 8.8.8.8 in there. No dice!!!

To wit, I now have an /etc/resolv.conf.pacnew file that is empty:

➜  ~ cat /etc/resolv.conf.pacnew 
# Resolver configuration file.
# See resolv.conf(5) for details.

Does this mean that /etc/resolv.conf is obsolete / useless now? If so, sad to see a simple solution go away, since it was working fine, and things that ain't broke don't need fixed.

2.) Okay, DNS resolution config is now a black box, lovely. Still, I managed to find out a workaround in /etc/nsswitch.conf:

# hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns
hosts: mymachines resolve files myhostname dns

What does this do? Why does this even work? I only tried it because someone told me it would fix it. It did, but now I want to know why? I don't even know what DNS server I'm using now. NetworkManager isn't touching /etc/resolv.conf, so I assume my DHCP-provided addresses aren't being used? Nor do I see a place to override them. Is NetworkManager GUI DHCP override being plumbed down in the system somehow in a mysterious way that doesn't modify the familiar /etc/resolv.conf now?

Why did we go from easy battle-tested stable DNS config to something scattered across multiple files and services that doesn't play well with NetworkManager?

Can somebody please shed some light on this?

Last edited by TheChuckster (2024-04-11 17:31:33)

-thc

Member

Registered: 2017-03-15

Posts: 510

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

nsswitch.conf works like this:

hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns
  |        |         |           |            |       |       |
host name resolution |           |            |       |       |
           |         |           |            |       |       |
          ask nss-mymachines/systemd-machined first - useful for containers/vms
                     |           |            |       |       |
                    ask nss-resolve/systemd-resolved  |       |
                                 |            |       |       |
                                if systemd-resolved is not unavailable, stop here and return the query result
                                              |       |       |
                                             look into the local files (e.g. /etc/hosts)
                                                      |       |
                                                     ask nss-myhostname
                                                              |
                                                             look into /etc/resolv.conf and query those nameservers

If a file like "/etc/resolv.conf.pacnew" exists it just means you installed a package that contained this file but "/etc/resolv.conf" was already there. The package configuration simply doesn't overwrite existing files.

If you can use "dig" you have installed "bind" - correct? "drill" does the same job and is installed in the Arch "base" set.

If you use "dig"/"drill" the query answer always contains a "SERVER: ..." field telling you which DNS server provided that answer.

Regarding your phenomena I suspect a misconfigured setup with systemd-resolved and NetworkManager - or a confilct with another network management.
Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-45s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

and please clarify what you want your network setup to be: DHCP with DNS or without DNS.

TheChuckster

Member

Registered: 2013-07-19

Posts: 14

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

Thanks for your help, -thc, also a Half-Life fan!

I would prefer NetworkManager to control my DNS servers on a per-network basis. To that end, I want the settings in the NetworkManager UI to be respected EXACTLY AS SHOWN. If I don't specify ::1, and it's also not sent via DHCP, it should not be used for DNS. There's a reason I want this flexibility: At home, I run my own local LAN DNS server on my router that, but when I'm on the go, I need to use the DHCP-provided servers from wifi networks, and sometimes I even want to override them for specific networks to DNS servers of my own preference. This was the original / working behavior, and I would like to go back to it; i.e. always follow the NetworkManager UI -- for each and every network shown in the list, if it says DHCP is used, use DHCP; if it has static overrides, use the overrides instead.

I do run several Arch machines at home, and I ended up encountering the same problem with all of them and having to manually remove ` [!UNAVAIL=return]` from each of their nsswitch configs, so I doubt it's some kind of one-off issue specific to my machine.

(base) ➜  ~ find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-45s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
avahi-daemon.service                          | multi-user.target.wants
avahi-daemon.socket                           | sockets.target.wants
cups.path                                     | multi-user.target.wants
cups.service                                  | multi-user.target.wants
cups.service                                  | printer.target.wants
cups.socket                                   | sockets.target.wants
dbus-org.freedesktop.Avahi.service            | system
dbus-org.freedesktop.nm-dispatcher.service    | system
dbus-org.freedesktop.resolve1.service         | system
display-manager.service                       | system
docker.service                                | multi-user.target.wants
gcr-ssh-agent.socket                          | sockets.target.wants
getty@tty1.service                            | getty.target.wants
gnome-keyring-daemon.socket                   | sockets.target.wants
ipp-usb.service                               | multi-user.target.wants
lm_sensors.service                            | multi-user.target.wants
NetworkManager.service                        | multi-user.target.wants
NetworkManager-wait-online.service            | network-online.target.wants
nfs-server.service                            | multi-user.target.wants
nmb.service                                   | multi-user.target.wants
ntpdate.service                               | multi-user.target.wants
openrgb.service                               | multi-user.target.wants
p11-kit-server.socket                         | sockets.target.wants
pipewire.socket                               | sockets.target.wants
pulseaudio.socket                             | sockets.target.wants
remote-fs.target                              | multi-user.target.wants
smbnetfs.service                              | multi-user.target.wants
smb.service                                   | multi-user.target.wants
sshd.service                                  | multi-user.target.wants
systemd-resolved.service                      | sysinit.target.wants
xdg-user-dirs-update.service                  | default.target.wants

*** Thanks again for taking a look!!

Last edited by TheChuckster (2024-04-11 17:30:11)

Lone_Wolf

Forum Moderator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 11,972

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

Please use [ code ] [ /code ] tags for such output, see https://bbs.archlinux.org/help.php#bbcode

NetworkManager.service                        | multi-user.target.wants

systemd-resolved.service                      | sysinit.target.wants

If systemd-resolved.service is active, NM defaults to using that for DNS . systemd-resolved has many weird quirks, one of which is that it replaces /etc/resolv.conf with a symlink .

Check https://wiki.archlinux.org/title/Systemd-resolved#DNS and https://wiki.archlinux.org/title/Networ … management .

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

(A works at time B)  && (time C > time B ) ≠  (A works at time C)

-thc

Member

Registered: 2017-03-15

Posts: 510

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

It's starting to make sense now. Your setup (NetworkManager with systemd-resolved for DNS management) is incomplete and you have to tell NSS to ignore the available systemd-resolved.

The first step is to restore systemd-resolved. Remove both "/etc/resolv.conf" and "/etc/resolv.conf.pacnew" and replace it with the correct "stub" link:

# ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

secondly restore nsswitch.conf:

hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns

and at last tell NetworkManager how to handle DNS by creating a file "dns.conf" in "/etc/NetworkManager/conf.d"

[main]
dns=systemd-resolved

Everything should be in working order then.

nl6720

The Evil Wiki Admin

Registered: 2016-07-02

Posts: 607

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

tell NetworkManager how to handle DNS by creating a file "dns.conf" in "/etc/NetworkManager/conf.d"

[main]
dns=systemd-resolved

That's not necessary.

systemd-resolved will be used automatically if /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, /run/systemd/resolve/resolv.conf or /usr/lib/systemd/resolv.conf.

-thc

Member

Registered: 2017-03-15

Posts: 510

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

That's not necessary.

I know.

TheChuckster

Member

Registered: 2013-07-19

Posts: 14

Re: glibc DNS failing, dig works fine, /etc/resolv.conf now IGNORED?!

Thanks everyone, any reason this suddenly "broke"? I've had these three Arch installs for years.

I edited my posts replacing the (unrecognized) Markdown with old school BBCode.

Also, @Lone_Wolf RE your signature, in propositional logic, the better symbol to use is the "does not imply" symbol (⇏) not the "unequal" symbol (≠). There's a whole notation around this.

Last edited by TheChuckster (2024-04-11 17:37:55)