`sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

Torxed · 2017-07-19 22:30:10

I'm starting to think this is an isolated problem to Arch.
The problem is that my machine has a lot of connections hording up `TIME_WAIT` sockets.
To get rid of this problem (while being aware of the risks), I set:

sysctl -w net.ipv4.tcp_fin_timeout=1

Normally sockets clear up like flies after a second.
But under Arch, they stay there for a minute despite the setting being set.

It's a permanent setting under `/etc/sysctl.d/10-timewait.conf`, but even that fails to do any good.
(yes, checking `sysctl net.ipv4.tcp_fin_timeout` show's the value is 1, but it has no actual affect on the system)

I've even tried to change the congestion control by doing:

sysctl -w net.ipv4_tcp_available_congestion_control=htcp

Because I thought this might be a setting only htcp honors. but it's the same result as with cubic and reno.
Why isn't Arch honoring the custom `tcp_fin_timeout`?

Last edited by Torxed (2017-07-19 22:42:59)

HiImTye · 2017-07-20 04:20:07

I think that the issue you need to address is what is causing that many timewait sockets. from your server fault post, it seems that the number of them is in the tens of thousands. that speaks to some sort of flooding, and simply reducing the duration of connections is not going to decongest your network.

Torxed · 2017-07-20 06:39:58

Correct HilmTye, it's the two sockets.
But that's by design, they're created to test the maximum I/O of the socket layer.

The only thing the server does, is:

for (;;)
    client_sock, client_addr = master_sock.accept()
    client_sock.close()

and the client does the same:

for (;;)
    s = socket()
    s.connect(('127.0.0.1', 1234))
    s.close()

Normally on a modern machine you should get roughly 20-40k connections per second without any major issues.
But for some reason, Arch can't be tweaked to forcibly close the `TIME_WAIT` sockets?

R00KIE · 2017-07-20 08:04:01

Arch does not do anything special to the kernel, check the PKGBUILD yourself. That said it is possible the cause is some other kernel configuration arch is using so you should check that. If the kernel configuration does not have anything "wrong" then you should be compiling your own kernel from git and if it is still broken complain upstream.

Torxed · 2017-07-20 09:17:17

R00KIE wrote:

Arch does not do anything special to the kernel, check the PKGBUILD yourself. That said it is possible the cause is some other kernel configuration arch is using so you should check that. If the kernel configuration does not have anything "wrong" then you should be compiling your own kernel from git and if it is still broken complain upstream.

I don't want to be rude, but my entire question is based around "some other kernel configuration arch is using".
I have no idea where to start looking for such a configuration change.

The only place I've found that could have had such a change, would have been https://git.archlinux.org/svntogit/pack … 56b91#n158

But there's not much going on in the kernel build really that's suspicious.
Again, I'm out of ideas where to look or what to look for.

Edit:
Would be neat with a list of customization that Arch does.
Perhaps a wiki stating all the modifications Arch does to the kernel for instance.
And maybe, just maybe point to where those changes are made.

Last edited by Torxed (2017-07-20 09:45:26)

R00KIE · 2017-07-20 09:52:55

If you are looking at line 158 in the PKGBUILD you are looking in the wrong place.

You want to look at the kernel configuration either in the files config.{i686,x86_64} depending on which architecture you are using or the configuration file of the currently running kernel in '/proc/config.gz'.

I assumed that if you are concerned about performance and doing benchmarks you would know all this and might even be compiling your own kernel with some configuration tweaks for squeezing as much performance as possible. That said, if this is a recent change you can look through the changes the kernel maintainers have done to the configuration and if that doesn't turn up anything interesting then bisect the kernel and try to find the commit that breaks things.

As I've said before, if there is no obvious change in the kernel configuration that explains the change in behavior you want to be talking to upstream about this.

Torxed · 2017-07-20 10:09:10

Cheers R00KIE, I'll have a look through the config.x86_64 and see if I find anything that sticks out.
I've never really had to compile a custom kernel with changed settings, only custom modules loaded. At least not in a year or two.

And I wasn't worried about benchmarks up until I noticed a considerable difference between Arch, Debian and Slack.
So this got me curious, why the difference and where is it? (It's a ~20-50K/second difference between them)

seth · 2017-07-20 15:57:20

Tried behavior on the LTS kernel?

Torxed · 2017-07-20 17:09:09

seth: Not yet, didn't know Arch had one til this morning.

[torxed@Daylight ~]$ diff config_arch config_working | wc -l
6452

This is a whole lot of changes to go through without knowing what to look for.

Last edited by Torxed (2017-07-20 17:13:42)

loqs · 2017-07-20 19:12:45

Build the linux package with config_working ( is config_working a 4.11 config as well ?) check if the resulting kernel has the issue?

Torxed · 2017-07-20 20:36:10

loqs wrote:

Build the linux package with config_working ( is config_working a 4.11 config as well ?) check if the resulting kernel has the issue?

Unfortunately `config_working` is from another distro, and might not work because the difference in modules?

Last edited by Torxed (2017-07-20 20:37:33)

loqs · 2017-07-20 20:52:23

The config should control which modules are built among other options. Options the kernel build system does not recognize will be ignored and missing options you will be prompted for (unless you run make olddefconfig).

Torxed · 2017-07-21 09:56:02

loqs wrote:

The config should control which modules are built among other options. Options the kernel build system does not recognize will be ignored and missing options you will be prompted for (unless you run make olddefconfig).

Tried to compile 4.8.6 but ending up with an error:

  HOSTCC  Documentation/mic/mpssd/mpssd.o
In file included from /usr/include/sys/types.h:215:0,
                 from /usr/include/stdlib.h:291,
                 from Documentation/mic/mpssd/mpssd.c:23:
Documentation/mic/mpssd/mpssd.c:93:10: error: initializer element is not constant
   .num = htole16(MIC_VRING_ENTRIES),
          ^
Documentation/mic/mpssd/mpssd.c:93:10: note: (near initialization for ‘virtcons_dev_page.vqconfig[0].nu’)
Documentation/mic/mpssd/mpssd.c:96:10: error: initializer element is not constant
   .num = htole16(MIC_VRING_ENTRIES),
          ^
Documentation/mic/mpssd/mpssd.c:96:10: note: (near initialization for ‘virtcons_dev_page.vqconfig[1].nu’)
Documentation/mic/mpssd/mpssd.c:113:10: error: initializer element is not constant
   .num = htole16(MIC_VRING_ENTRIES),
          ^
Documentation/mic/mpssd/mpssd.c:113:10: note: (near initialization for ‘virtnet_dev_page.vqconfig[0].nu’)
Documentation/mic/mpssd/mpssd.c:116:10: error: initializer element is not constant
   .num = htole16(MIC_VRING_ENTRIES),
          ^
Documentation/mic/mpssd/mpssd.c:116:10: note: (near initialization for ‘virtnet_dev_page.vqconfig[1].nu’)
Documentation/mic/mpssd/mpssd.c:119:19: error: initializer element is not constant
  .host_features = htole32(
                   ^
Documentation/mic/mpssd/mpssd.c:119:19: note: (near initialization for ‘virtnet_dev_page.host_features’)
Documentation/mic/mpssd/mpssd.c:145:10: error: initializer element is not constant
   .num = htole16(MIC_VRING_ENTRIES),
          ^
Documentation/mic/mpssd/mpssd.c:145:10: note: (near initialization for ‘virtblk_dev_page.vqconfig[0].nu’)
Documentation/mic/mpssd/mpssd.c:148:3: error: initializer element is not constant
   htole32(1<<VIRTIO_BLK_F_SEG_MAX),
   ^
Documentation/mic/mpssd/mpssd.c:148:3: note: (near initialization for ‘virtblk_dev_page.host_features’)
Documentation/mic/mpssd/mpssd.c:150:14: error: initializer element is not constant
   .seg_max = htole32(MIC_VRING_ENTRIES - 2),
              ^
Documentation/mic/mpssd/mpssd.c:150:14: note: (near initialization for ‘virtblk_dev_page.blk_config.seg_max’)
Documentation/mic/mpssd/mpssd.c:151:15: error: initializer element is not constant
   .capacity = htole64(0),
               ^
Documentation/mic/mpssd/mpssd.c:151:15: note: (near initialization for ‘virtblk_dev_page.blk_config.capacity’)
make[3]: *** [scripts/Makefile.host:124: Documentation/mic/mpssd/mpssd.o] Error 1
make[2]: *** [scripts/Makefile.build:440: Documentation/mic/mpssd] Error 2
make[1]: *** [scripts/Makefile.build:440: Documentation/mic] Error 2
make: *** [Makefile:928: vmlinux_prereq] Error 2

Trying to find information on this or if I messed up one of the 130 options I just defaulted when running make.
Following this guide: https://wiki.archlinux.org/index.php/Ke … ompilation

loqs · 2017-07-21 12:00:04

Do you have a working config for a kernel upstream is still supporting 4.4.y, 4.9.y, 4.11.y, 4.12.y, 4.13-rcx?
The 130 new options seems strange to me if the config was generated for the 4.8.6, possibly an architecture mismatch?
You could probably work around the error by just altering Documentation/Makefile to remove mic see https://github.com/torvalds/linux/commi … 394b4fd028
but it feels wrong to me as the same code is still there under samples https://git.kernel.org/pub/scm/linux/ke … sd/mpssd.c

Arch Linux

#1 2017-07-19 22:30:10

`sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#2 2017-07-20 04:20:07

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#3 2017-07-20 06:39:58

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#4 2017-07-20 08:04:01

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#5 2017-07-20 09:17:17

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#6 2017-07-20 09:52:55

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#7 2017-07-20 10:09:10

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#8 2017-07-20 15:57:20

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#9 2017-07-20 17:09:09

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#10 2017-07-20 19:12:45

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#11 2017-07-20 20:36:10

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#12 2017-07-20 20:52:23

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#13 2017-07-21 09:56:02

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

#14 2017-07-21 12:00:04

Re: `sysctl -w net.ipv4.tcp_fin_timeout=1` has no affect?

Board footer