[SOLVED] docker 17.04 disables IPv6 on the host

haggy · 2017-04-14 19:50:58

Hello fellow Archers,

since a few weeks i noticed delays when ssh-ing into my server which is running an up-to-date Archlinux. After investigating a bit i found out, that "something" disables ipv6 in the kernel after reboot, leading to no IPv6 addresses on the NICs, eventually causing delays because i'm connecting from an IPv6 enabled laptop which preferes v6 over v4 (fallback). The question is, what disables IPv6 and how can i find out? I never disabled it by myself and did not find any traces of sysctl calls or settings to disable_ipv6 neither on the system nor the kernel boot line. Manually enabling it again after reboot (disable_ipv6 = 0) brings up the inet6 links immediately.

Any clues? I'm using Kernel 4.10.9 and systemd-networkd. Could be the kernel or a recent systemd upgrade. networkd manpage mentions that it'll automatically disable ipv6 if certain conditions are not met, but i already checked all of them and this does would explain ipv6 getting disabled globally for any interface, even the ones not managed by networkd.

Thanks
Haggy

EDIT: Seems like docker 17.04 is the root cause for this issue - fixed on next release: https://github.com/moby/moby/issues/32433

Last edited by haggy (2017-05-11 11:47:56)

graysky · 2017-04-14 21:36:46

I assume you verified that none of the methods listed on the wiki are to blame?

haggy · 2017-04-15 07:31:23

Exactly, because i never intended to disable IPv6 - i need it to reach the box from the Internet.

olive · 2017-04-15 07:44:35

You can disable ipv6 by passing "ipv6.disable=1" to the kernel command line (edit your bootloader configuration file).

As for the explanation why it could be useful. The delays are often due to something that does not respond. If ipv6 is enabled but not correctly functioning, it can happen that there is an ipv6 query that does not responds, the system waits a few second before deciding it has failed and try ipv4 which works correctly. Because there are still no (AFAIK) ipv6 only things or website, all will work but with a delay.

Note that the problematic thing is not necessarily the computer itself. I remember (some years ago) an ADSL modem/router, this ADSL modem router gave its own address as DNS by DHCP (like 192.168.1.2) and relay the calls to the real DNS. But when the system made an ipv6 DNS query, the ADSL modem did not understand it and does not respond at all. Then the system waits for a few seconds before making an ipv4 query that works.

This specific problem has been solved several years ago for me, but I would not be astonished that some kind of buggy routers lie in the path of some people.

haggy · 2017-04-15 07:51:35

Thanks olive, however my problem is not that i want to disable IPv6, but that it disabled itself for some reason. I want and need IPv6, and every other box in my network reliably gets a prefix assigned and works flawlessly, as did my server until a few weeks ago. Since then, after reboot IPv6 is disabled via sysctl and i don't find any reason or traces for this. Manually re-enabling via sysctl brings up the IPv6 connections immediately. I know about the reasons for the delay - that is also not my problem, but it led me to the actual problem of IPv6 being disabled on the server without intent.

olive · 2017-04-15 08:03:42

There are nothing in /etc/sysctl.d? The usual solution to find out is to try an "incremental search" until you find the culprit, for example:

1) pass the "break" option to the kernel command line: this will drop you in the initrd, thus at a very early stage. Look if ipv6 is disabled.
2) Disable any high-level network configuration tool or service (like systemd-networkd, network-manager, etc...). Log in to the console (or with a plain WM that does not do anything with the internet) and try to configure the network by hand. Look if it is still disabled.

Try also to enable via some file in /etc/sysctl.d (for example /etc/sysctl.d/99-testipv6.conf).

You normally should find the culprit and investigate further.

haggy · 2017-04-15 08:08:34

Thanks for the suggestions - will try the "break" option and go on from there.

shervinkh · 2017-04-17 18:26:52

Happened to me too.
Did an update and kernel went from 4.10.3 (or so) to 4.10.9 and then after the reboot all of a sudden ipv6 was disabled and I had to enable it via sysctl.

haggy · 2017-04-18 05:24:17

Thanks shervinkh, sounds like a kernel-related issue then. Will file a report.
Maybe worth to check if we share a common setup - i'm running systemd-networkd, systemd-bootd and Docker which might interfere with the network stack. Anything related on your side?

My networkd-conf is nothing special:

[Match]
Name=enp0s31f6

[Network]
DHCP=ipv4
Domains=fritz.box

[DHCP]
UseDomains=true

Note that i already tried a lot of settings without any effect on the sysctl setting.

dmesg: https://clbin.com/D9220?hl

haggy · 2017-04-18 05:31:49

https://bugs.archlinux.org/task/53702

loqs · 2017-04-18 11:38:19

When reporting the bug you appear to have missed Reporting_bug_guidelines#Upstream_or_Arch.3F

If Arch is not responsible for a bug, the problem will not be solved by reporting the bug to Arch developers. Responsibility for a bug is said to lie upstream when it is not caused through the distribution's porting and integration efforts.

See also https://bugs.archlinux.org/task/53664#comment156789 Bisecting_bugs and reporting-bugs

haggy · 2017-04-18 12:11:14

Thanks, i know about the guidelines. However it's not easy to decide whether this is an upstream or Arch bug.

loqs · 2017-04-18 17:34:48

@haggy If you could verify this

shervinkh wrote:

Did an update and kernel went from 4.10.3 (or so) to 4.10.9 and then after the reboot all of a sudden ipv6 was disabled and I had to enable it via sysctl.

Then the only change that has not been the application of upstream patches is https://git.archlinux.org/svntogit/pack … db9a2bdb08 which seems a very unlikely cause
If you do not have 4.10.3-1 in your package cache you can use Arch_Linux_Archive to obtain it. Assuming you verify shervinkh's finding you could find which linux package release the issue occurs with and bisect between that and the last good one.

haggy · 2017-04-19 06:57:17

Last known working kernel for me was 4.10.8 i think - will try that one first.

haggy · 2017-04-19 07:56:37

OK, tried 4.10.8, 4.10.6 and finally 4.10.3 - i can safely say now it's not a kernel issue as IPv6 is still disabled after rebooting any of these kernels. Must be some related system upgrade since then.

tom.ty89 · 2017-04-19 08:05:35

haggy wrote:

OK, tried 4.10.8, 4.10.6 and finally 4.10.3 - i can safely say now it's not a kernel issue as IPv6 is still disabled after rebooting any of these kernels. Must be some related system upgrade since then.

I am not seeing this issue on the latest Arch ISO (4.10.6) either. Have you tried disabling all network agent you use and see if it is still disabled on boot?

graysky · 2017-04-19 08:08:51

Just for kicks, can you install a fresh Arch to a spare partition and verify that IPv6 is disabled there? If it is, then:

graysky wrote:

I assume you verified that none of the methods listed on the wiki are to blame?

haggy · 2017-04-19 08:48:37

Still have to disable networkd - unfortunately i'm not at home and disabling via ssh would probably lock me out because noone configures the nic manually then. Will update you.

shervinkh · 2017-04-20 16:02:35

haggy wrote:

Thanks shervinkh, sounds like a kernel-related issue then. Will file a report.
Maybe worth to check if we share a common setup - i'm running systemd-networkd, systemd-bootd and Docker which might interfere with the network stack. Anything related on your side?

I only run docker from those three.

I'm not sure if it was kernel or NetworkManager or anything else but the system upgrade caused it and currently i have ipv6 disabled by default after each reboot which is a pain for me because i use ipv6 vpn.

I haven't got the time to dive into the logs and configs yet.

haggy · 2017-04-20 20:44:56

Ok, got the culprit of this issue - it's my Plex Docker container which is running in privileged mode (as suggested by Plex people). For reasons unknown it immediately disables ipv6 in the kernel for all interfaces when it starts up. Disabling the Plex container leaves my inet6 addresses configured, but as soon as i start the container, they disappear from all interfaces (even lo!).

Have to investigate and nag the Plex people... IPv6 support in Plex itself is enabled, but that does not seem to help.

@shervinkh: You're not running https://hub.docker.com/r/plexinc/pms-docker/ by chance?

shervinkh · 2017-04-21 13:30:25

I've also found the issue in a docker container.

I didn't have that plex container. It was the "cassandra database" container that was causing the issue for me. and unfortunately it was set to start as soon as the docker daemon starts. Starting that would set net.conf.ipv6.all.disable_ipv6=1 for all,default and each network interface which is very weird.

I still don't know whether it's a docker daemon bug or specific docker container bug.

haggy · 2017-04-24 07:22:26

Well at least for me it's definitely plex running in privileged mode. As soon as the container starts (doesn't matter if it autostarts with docker, happens also wenn i start it manually at later time), IPv6 is gone.

yochaigal · 2017-04-29 19:55:06

I think I'm experiencing the same thing. I'm on 4.10.11, I've never set sysctl configuration parameters to disable ipv6, or GRUB. Yet ipv6 is definitely disabled, and /proc/sys/net/ipv6 does not exist. I'm using Network Manager, and it is not configured to disable ipv6 in any way.
Am I missing something, or am I experiencing the same issue as haggy?

Thanks

haggy · 2017-04-30 11:55:51

You're also using Docker?

yochaigal · 2017-04-30 12:14:19

No, not using any containers. I do use Plex, but not in that way.

Arch Linux

#1 2017-04-14 19:50:58

[SOLVED] docker 17.04 disables IPv6 on the host

#2 2017-04-14 21:36:46

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#3 2017-04-15 07:31:23

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#4 2017-04-15 07:44:35

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#5 2017-04-15 07:51:35

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#6 2017-04-15 08:03:42

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#7 2017-04-15 08:08:34

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#8 2017-04-17 18:26:52

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#9 2017-04-18 05:24:17

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#10 2017-04-18 05:31:49

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#11 2017-04-18 11:38:19

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#12 2017-04-18 12:11:14

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#13 2017-04-18 17:34:48

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#14 2017-04-19 06:57:17

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#15 2017-04-19 07:56:37

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#16 2017-04-19 08:05:35

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#17 2017-04-19 08:08:51

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#18 2017-04-19 08:48:37

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#19 2017-04-20 16:02:35

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#20 2017-04-20 20:44:56

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#21 2017-04-21 13:30:25

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#22 2017-04-24 07:22:26

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#23 2017-04-29 19:55:06

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#24 2017-04-30 11:55:51

Re: [SOLVED] docker 17.04 disables IPv6 on the host

#25 2017-04-30 12:14:19

Re: [SOLVED] docker 17.04 disables IPv6 on the host

Board footer