Stale DNS servers in resolv.conf with netctl

yuvadm · 2017-03-12 09:21:29

For all my networking purposes I use netctl so connect to both wireless networks, as well as to wired networks, for that I usually use a profile which is essentially the ethernet-dhcp example profile.

Usually netctl works great, but there's one problem I have when connecting to a specific corporate network. I can connect to it without a problem using the usual ethernet DHCP profile

# netctl start dhcp

after which I can see that:

$ cat /etc/resolv.conf
domain corp.example.com
nameserver 1.2.3.4
nameserver 5.6.7.8

So far so good. When I'm done on that network I run

# netctl stop-all

. Then sometime after I connect to a new network, say an existing wireless profile,

# netctl start home

and then I see this:

$ cat /etc/resolv.conf
domain corp.example.com
nameserver 1.2.3.4
nameserver 5.6.7.8
nameserver 192.168.1.1

So 192.168.1.1 is the expected local router/DNS, but I also see the stale DNS servers from the corporate net. This creates a ~20 second delay in all DNS queries since those nameservers do not exist on the net.

What's even weirder is that I can manually delete the first three stale lines, but they'll always come back after a suspend/resume cycle (this is a laptop).

I don't seem to have any other DNS resolvers running on the system, so why these DNS servers keep showing up in resolv.conf is beyond me. Ideas?

Last edited by yuvadm (2017-03-12 09:23:17)

Awebb · 2017-03-12 09:41:02

The content of resolv.conf is generated by your dhcp solution. I use dhcpcd, it doesn't do this. What's yours?

brebs · 2017-03-12 09:41:53

I recommend Unbound.

yuvadm · 2017-03-12 10:06:36

netctl seems to use dhcpcd as well, how can I debug this behavior?

graysky · 2017-03-12 11:32:56

Slightly OT but for a simple dhcp setup, why use netctl at all? Just use the included systemd-networkd: https://wiki.archlinux.org/index.php/Systemd-networkd

yuvadm · 2017-03-12 12:00:23

True, I use systemd-networkd sometimes on simple setups, but when I also need various wifi networks i prefer to just use netctl for everything.

rsmarples · 2017-03-13 15:26:02

FULL DISCLAIMER: I maintain dhcpcd, I have an obvious bias.

When a DHCP client exits, the default behavior is to persist the current network config. This could explain the stale data.
Maybe you need to RELEASE the DHCP lease when exiting, which will remove the config when the client exits.
With dhcpcd you can add the command `release` to /etc/dhcpcd.conf to get this to happen by default, or use the -k directive instead of -x to cause dhcpcd to exit.

But really, using something like netctl can make this scenario more complex than it needs to be.
You can use dhcpcd to manage the connection for you, it will listen to many kernel events such as wired plugged in, pulled out, or a change of wireless network.
There is no need to manually change via a profile, just configure wpa_supplicant.conf for each of your networks and let it roll. dhcpcd will listen to it.

Arch even has a fairly accurate wiki on dhcpcd: https://wiki.archlinux.org/index.php/dhcpcd

Need static stuff for different networks? Sure, dhcpcd can do that.
Need to ARPING a static gateway to work out which network profile you want? Sure, dhcpcd can do that.
Seamlessly transition between wired -> wireless -> wired? Sure, dhcpcd can do that.
Need to make a cup of tea? OK, dhcpcd cannot do that, but it will handle any kind of network configuration you want.
The only thing it won't do is actually setup the link (VPN, SSID passwords, etc) because that's outside the scope of dhcpcd.

Last edited by rsmarples (2017-03-13 15:26:34)

yuvadm · 2017-03-13 15:38:51

Thanks for chiming in @rsmarples!

I might try to use dhcpcd exclusively if indeed it simplifies things. And for now I might use `release` in dhcpcd.conf.

But what I don't understand is that even if I use `persist` (as it is by default) why does dhcpcd keep the stale nameservers and not override them with the new ones? Do you think this is an upstream bug?

rsmarples · 2017-03-19 21:45:07

Good question!
Truthfully, I don't know, it depends on things.

dhcpcd shouldn't keep any stale info around IF it's running in master mode (ie, under it's own control).
If it does, that's a bug and I'll work with you through that to fix it (plz open a task here).

If it's under control of a 3rd party, like say netctl then yes it can leave stale information if netctl asks dhcpcd to exit on wired and then start on wireless because it's not been released.
Releasing has other ramifications though, so it's not recommended by default.

thorsten · 2017-05-22 10:09:32

The file /etc/resolv.conf is managed by the tool resolvconf. The resolvconf tool manages the name server for each interface separately (in /run/resolvconf/interfaces) and merges them into a single /etc/resolv.conf (see man resolvconf).

The issue is, that dhcpcd never tells resolvconf when an interface shuts down. (That is either due to a bug in dhcpcd or due to netctl not telling dhcpcd to do so, I don't know). In order to shut down a interface, e.g. eth0, one needs to call resolvconf -d eth0.dhcp. Wildcards are also supported, so resolvconf -d 'eth0.*' works as well.

TLDR/Workaround: In order to shut down all interfaces for resolvconf, one can call resolvconf -d '*.*' after each netctl stop-all to be absolutely sure that all data is cleand up properly.

rsmarples · 2017-05-22 10:24:20

thorsten wrote:

The issue is, that dhcpcd never tells resolvconf when an interface shuts down. (That is either due to a bug in dhcpcd or due to netctl not telling dhcpcd to do so, I don't know). In order to shut down a interface, e.g. eth0, one needs to call resolvconf -d eth0.dhcp. Wildcards are also supported, so resolvconf -d 'eth0.*' works as well.

I'm author of both dhcpcd and openresolv.
Please define how an interface "shuts down".

If you mean carrier is lost, interface is removed, interface is brought down (say ifconfig eth0 down) or dhcpcd is instructed to release then dhcpcd will instruct resolvconf to delete the interface DNS via the 20-resolv.conf hook.
If none of the above are true then the DNS is expected to persist for the interface.

Observe

# Delete interface DNS
resolvonf -d "eth0.*"
# Check it's not there
resolvonf -l "eth0.*"
# Start dhcpcd
dhcpcd eth0
# Check DNS is there
resolvonf -l "eth0.*"
# Instruct dhcpcd to release (you could do `ifconfig eth0 down` here as well)
dhcpcd -k eth0
# Check DNS is now gone
resolvonf -l "eth0.*"

thorsten · 2017-05-22 11:39:15

Thanks for your quick reply!

rsmarples wrote:

# Delete interface DNS
resolvonf -d "eth0.*"
# Check it's not there
resolvonf -l "eth0.*"
# Start dhcpcd
dhcpcd eth0
# Check DNS is there
resolvonf -l "eth0.*"
# Instruct dhcpcd to release (you could do `ifconfig eth0 down` here as well)
dhcpcd -k eth0
# Check DNS is now gone
resolvconf -l "eth0.*"

Indeed, the DNS is gone after this sequence. However the issue seems to be that netctl does not stop dhcpcd as in your listing, but simply via SIGTERM.

If I stop a netctl profile, then the following messages are logged:

$ journalctl -f --unit=system-netctl.slice
Mai 22 13:21:25 ghul dhcpcd[18727]: sending signal TERM to pid 18263
Mai 22 13:21:25 ghul dhcpcd[18263]: received SIGTERM, stopping
Mai 22 13:21:25 ghul dhcpcd[18263]: eth0: removing interface
Mai 22 13:21:25 ghul dhcpcd[18727]: waiting for pid 18263 to exit
Mai 22 13:21:25 ghul dhcpcd[18263]: dhcpcd exited

And indeed if I do the following manually, then the DNS entry survives:

$ resolvconf -l 'eth0.*'
No resolv.conf for interface eth0.*
$ ifconfig eth0 up
$ dhcpcd eth0
DUID 00:01:00:01:15:5d:43:9f:00:1f:16:1c:a1:89
eth0: IAID 16:1c:a1:89
eth0: adding address fe80::89ae:eb6a:37c4:3fbe
if_addaddress6: Permission denied
eth0: soliciting an IPv6 router
eth0: rebinding lease of 192.168.1.10
eth0: probing address 192.168.1.10/24
eth0: leased 192.168.1.10 for 600 seconds
eth0: adding route to 192.168.1.0/24
eth0: adding default route via 192.168.1.1
forked to background, child pid 19359
$ kill -TERM 19359
$ ps aux|grep dhcpcd
root     19401  0.0  0.1  13188  2476 pts/8    S+   13:25   0:00 grep --color=auto dhcpcd
$ resolvconf -l 'eth0.*'
# resolv.conf from eth0.dhcp
# Generated by dhcpcd from eth0.dhcp
domain informatik.uni-erlangen.de
search informatik.uni-erlangen.de
nameserver 131.188.0.11
$ ls /run/resolvconf/interfaces/
eth0.dhcp

So I see two solutions:

dhcpcd must treat SIGTERM as it does -k and leave resolvconf in a clean state (downside: dhcpcd shutdown takes longer in case of SIGTERM)
netctl must stop dhcpcd via dhcpcd -k (downside: netctl becomes less generic)

I would prefer the first one (but I'm not an expert on network tools...)

rsmarples · 2017-05-22 13:21:39

thorsten wrote:

So I see two solutions:
dhcpcd must treat SIGTERM as it does -k and leave resolvconf in a clean state (downside: dhcpcd shutdown takes longer in case of SIGTERM)
netctl must stop dhcpcd via dhcpcd -k (downside: netctl becomes less generic)
I would prefer the first one (but I'm not an expert on network tools...)

Speaking as dhcpcd upstream, option 1 isn't happening.
SIGTERM is `dhcpcd -x` - just stop
SIGALRM is `dhcpcd -k` - stop and remove configuration

However, this is also configurable by you.
dhcpcd.conf ships with persistent configuration enabled by default - just comment this out to get -x to behave like -k.

thorsten · 2017-05-22 13:59:49

rsmarples wrote:

SIGTERM is `dhcpcd -x` - just stop
SIGALRM is `dhcpcd -k` - stop and remove configuration
However, this is also configurable by you.
dhcpcd.conf ships with persistent configuration enabled by default - just comment this out to get -x to behave like -k.

And how should netctl stop dhcpcd if one wants to 1) keep the dhcp lease for future use and 2) have resolvconf in a sane state?

Or is this issue simply a bug in resolvconf? I'm wondering why resolvconf includes nameservers of interfaces that are deactivated.

Side remark: Right now, netctl stops dhcpcd either via -x or via -k, depending on the configuration option DHCPReleaseOnStop: https://git.archlinux.org/netctl.git/tr … dhcpcd#n22

Last edited by thorsten (2017-05-22 14:10:44)

rsmarples · 2017-05-22 14:09:44

So it is configurable, good.

And how should netctl stop dhcpcd if one wants to 1) keep the dhcp lease for future use and 2) have resolvconf in a sane state?

Bring the interface down (ifconfig eth0 down) before asking dhcpcd to stop with -x.

Arch Linux

#1 2017-03-12 09:21:29

Stale DNS servers in resolv.conf with netctl

#2 2017-03-12 09:41:02

Re: Stale DNS servers in resolv.conf with netctl

#3 2017-03-12 09:41:53

Re: Stale DNS servers in resolv.conf with netctl

#4 2017-03-12 10:06:36

Re: Stale DNS servers in resolv.conf with netctl

#5 2017-03-12 11:32:56

Re: Stale DNS servers in resolv.conf with netctl

#6 2017-03-12 12:00:23

Re: Stale DNS servers in resolv.conf with netctl

#7 2017-03-13 15:26:02

Re: Stale DNS servers in resolv.conf with netctl

#8 2017-03-13 15:38:51

Re: Stale DNS servers in resolv.conf with netctl

#9 2017-03-19 21:45:07

Re: Stale DNS servers in resolv.conf with netctl

#10 2017-05-22 10:09:32

Re: Stale DNS servers in resolv.conf with netctl

#11 2017-05-22 10:24:20

Re: Stale DNS servers in resolv.conf with netctl

#12 2017-05-22 11:39:15

Re: Stale DNS servers in resolv.conf with netctl

#13 2017-05-22 13:21:39

Re: Stale DNS servers in resolv.conf with netctl

#14 2017-05-22 13:59:49

Re: Stale DNS servers in resolv.conf with netctl

#15 2017-05-22 14:09:44

Re: Stale DNS servers in resolv.conf with netctl

Board footer