Arch Linux

beej

Member

From: Bend, OR

Registered: 2009-01-27

Posts: 22

Website

[SOLVED] named bind defunct

Solved: Either (A) upgrade to bind 9.11.0.P3-3 or later or (B) use the -L workaround, below.

Just did a reboot and DNS stopped working. For the record, I use systemctl, of course, but the problem seems to be more at the named level which is why I'm running it directly in some of these examples.

If systemd fires named up, the named process becomes defunct.

# systemctl start named

Mar 10 09:43:57 goat systemd[1]: Started Internet domain name server.
Mar 10 09:43:57 goat named[3587]: starting BIND 9.11.0-P3 <id:4801fbc>
Mar 10 09:43:57 goat named[3587]: running on Linux x86_64 4.10.1-1-ARCH #1 SMP P
Mar 10 09:43:57 goat named[3587]: built with '--prefix=/usr' '--sysconfdir=/etc'
Mar 10 09:43:57 goat named[3587]: running as: named -f -u named
Mar 10 09:43:57 goat named[3587]: ----------------------------------------------
Mar 10 09:43:57 goat named[3587]: BIND 9 is maintained by Internet Systems Conso
Mar 10 09:43:57 goat named[3587]: Inc. (ISC), a non-profit 501(c)(3) public-bene
Mar 10 09:43:57 goat named[3587]: corporation.  Support and training for BIND 9 
Mar 10 09:43:57 goat named[3587]: available at https://www.isc.org/support
Mar 10 09:43:57 goat named[3587]: ----------------------------------------------
Mar 10 09:43:57 goat named[3587]: adjusted limit on open files from 4096 to 1048
Mar 10 09:43:57 goat named[3587]: found 4 CPUs, using 4 worker threads
Mar 10 09:43:57 goat named[3587]: using 3 UDP listeners per interface
Mar 10 09:43:57 goat named[3587]: using up to 4096 sockets

# ps aux | grep named
named     3587  0.0  0.0      0     0 ?        Zsl  09:43   0:00 [named] <defunct>

And of course lookups do not work.

So I kill it and fire it up again trying to get more information. -d 11 doesn't give more information in journalctl... so I figure maybe I can use a different log file:

# /usr/bin/named -f -L /var/log/named.log -u named

... And it doesn't crash. Figures. -L anything seems to make it not crash, even if named doesn't have write permission to the log file location.

Since I don't think I have logrotate running for anything in /var/log any longer, I just did this:

# cat /etc/systemd/system/multi-user.target.wants/named.service
[Unit]
Description=Internet domain name server
After=network.target

[Service]
#ExecStart=/usr/bin/named -f -u named
ExecStart=/usr/bin/named -f -L /dev/null -u named
ExecReload=/usr/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Which at least has it working, but is pretty awful.

Any ideas on this one for where else to look for logs or leads? Thanks.

Last edited by beej (2017-03-18 04:03:27)

jason

Member

From: /dev/null

Registered: 2003-01-13

Posts: 102

Re: [SOLVED] named bind defunct

I'm having the same exact problem.  It just started after I restarted bind.

frakka

Member

Registered: 2017-03-11

Posts: 1

Re: [SOLVED] named bind defunct

The same for me, some minutes ago.
After full pacman update and reboot.

Many thanks for your suggestion: at least I was able to start it and it is working

ivanborodin

Member

Registered: 2013-08-30

Posts: 58

Re: [SOLVED] named bind defunct

Same situation here: working fine until today; full-system upgrade (bind now 9.11.0.P3-2), and after rebooting, named not working anymore: same log entries as in the previous post

amish

Member

Registered: 2014-05-10

Posts: 475

Re: [SOLVED] named bind defunct

Same problem for me.

named goes defunct immediately after start. (restarting does not help)

Had to change DNS from 127.0.0.1 to OpenDNS (208.67.222.222)

hkoch

Member

Registered: 2016-05-10

Posts: 4

Re: [SOLVED] named bind defunct

moi too. I was able to fix this using a systemd dropin with the same ExecStart line (keep in mind that in a dropin you have to empty the ExecStart entry first):

[Service]
ExecStart=
ExecStart=/usr/bin/named -f -L /dev/null -u named

You can create this file with

sudo systemctl edit named

.
If and when a fix to named comes out, simply

rm /etc/systemd/system/named.dropin.d/override.conf

.

1730@gmx.net

Member

Registered: 2017-03-12

Posts: 1

Re: [SOLVED] named bind defunct

Thanks, now I just would like to know what caused this. During my last upgrade named wasn't changed.

Thanks @hkoch and @beej for posting a workaround!

awado

Member

Registered: 2014-02-13

Posts: 10

Re: [SOLVED] named bind defunct

I think it's a bug of the bind binary in the pacman repos. I did a fresh install of Arch Linux with just "named" alone with exact the same issue. Seems the permissions for logging cause named to hang.

amish

Member

Registered: 2014-05-10

Posts: 475

Re: [SOLVED] named bind defunct

Actually its not hanging.. its dying and becoming zombie!

seblu

Developer

From: Paris

Registered: 2011-01-20

Posts: 9

Re: [SOLVED] named bind defunct

Downgrading glibc to 2.24 workarounds the problem on my servers.

doctorzeus

Member

Registered: 2011-12-24

Posts: 79

Re: [SOLVED] named bind defunct

Spent an entire day almost trying to figure this out on our office server (been using the gateway one until then).

Lifesaver thankyou!

Last edited by doctorzeus (2017-03-12 20:00:46)

papajonpizza

Member

From: Los Angeles

Registered: 2013-06-13

Posts: 12

Website

Re: [SOLVED] named bind defunct

Many thanks @hkoch and @beej!  Workaround is working great.
I noticed journalctl --all -f  is not showing the DNS lookups anymore but should be ok when the named is updated/fixed.

file I have to delete after the fix is released  -->   /etc/systemd/system/named.service.d/override.conf

Last edited by papajonpizza (2017-03-13 03:50:55)

---

Xiaomi Notebook Air 13 - 13.3" display | Intel Core i5 6200 | 8GB RAM | 256GB SSD | Arch Linux -  GNOME desktop

Enverex

Member

From: UK

Registered: 2007-06-13

Posts: 159

Website

Re: [SOLVED] named bind defunct

Christ, this was nasty. Thanks for the simple workaround hkoch.

Has anyone logged this against the Arch bug tracker?

severach

Member

Registered: 2015-05-23

Posts: 198

Re: [SOLVED] named bind defunct

I spent all day Sunday figuring out why one of my packages suddenly stopped working. Downgrading glibc was the solution. Downgrading is tricky because glibc interlocks with many other packages.

Multilib:
IgnorePkg   = gcc-multilib glibc lib32-glibc binutils valgrind gcc-libs-multilib lib32-gcc-libs
pacman -U gcc-multilib-6.3.1-1-x86_64.pkg.tar.xz glibc-2.24-2-x86_64.pkg.tar.xz lib32-glibc-2.24-2-x86_64.pkg.tar.xz binutils-2.27-1-x86_64.pkg.tar.xz valgrind-3.12.0-1-x86_64.pkg.tar.xz gcc-libs-multilib-6.3.1-1-x86_64.pkg.tar.xz lib32-gcc-libs-6.3.1-1-x86_64.pkg.tar.xz
 
Non Multilib:
IgnorePkg   = gcc glibc binutils valgrind gcc-libs
pacman -U gcc-6.3.1-1-x86_64.pkg.tar.xz glibc-2.24-2-x86_64.pkg.tar.xz binutils-2.27-1-x86_64.pkg.tar.xz valgrind-3.12.0-1-x86_64.pkg.tar.xz gcc-libs-6.3.1-1-x86_64.pkg.tar.xz

glibc was terminating my program claiming "stack smashing". Confusion ensues as all you ever see is [zombie] in the ps listing if even you're quick enough to catch it.

Looks like stack smashing [detection] has been turned up to ludicrous speed and it can't be shut off per package with compile flags. It's a good idea but getting to where Arch is reliable again is going to be painful, particularly if other distros don't jump on board.

Another Allanbrokeit moment.

Last edited by severach (2017-03-13 20:05:16)

loqs

Member

Registered: 2014-03-06

Posts: 18,163

Re: [SOLVED] named bind defunct

@Enverex https://bugs.archlinux.org/task/53240#comment156109
@severach search engine query for "stack smashing" limited to archlinux.org and the last 30 days only finds your post. The proposed fix for bind is not related to stack smashing.  Perhaps you should start a new topic on it.

hkoch

Member

Registered: 2016-05-10

Posts: 4

Re: [SOLVED] named bind defunct

@Enverex https://bugs.archlinux.org/task/53240#comment156109

oooh nice - looks like this will get some attention now. I love it when security breaks things

Thanks, @loqs !

loqs

Member

Registered: 2014-03-06

Posts: 18,163

Re: [SOLVED] named bind defunct

Should be fixed with bind-9.11.0.P3-3

hkoch

Member

Registered: 2016-05-10

Posts: 4

Re: [SOLVED] named bind defunct

works for me. After the upgrade I took out my override.conf file, then ran:

systemctl daemon-reload && systemctl restart named

and my named is once again running (and logging properly).

Thanks, everyone!

ivanborodin

Member

Registered: 2013-08-30

Posts: 58

Re: [SOLVED] named bind defunct

bind-9.11.0.P3-3 works for me too.

Thank you VERY much to all the ones involved in debugging/fixing this issue (and in the meantime for the clever workaround) ... (at least) too me, it was FAST.