[SOLVED] NAT Puzzle: What's going on here?

pgoetz · 2014-06-09 18:25:02

I recently ran into some very unexpected behavior involving package dependencies, iptables, and NAT. Just in case this is an issue, here is my network topology:

{Internet} --- Netgear WNDR3700 ------ (eno1)
router w/ NAT |
------------------
| Arch Server |
------------------
|
(enp2s0) ----------- Netgear GS105 ------------------ [Test machine]
Switch

and here are the profiles for the ethernet ports on the Arch server:

Description='A basic static ethernet connection'
Interface=eno1
Connection=ethernet
IP=static
Address=('192.168.1.5/24')
#Routes=('192.168.0.0/24 via 192.168.1.2')
Gateway='192.168.1.1'
DNS=('192.168.1.1')

Description='A basic static ethernet connection'
Interface=enp2s0
Connection=ethernet
IP=static
Address=('172.18.90.1/24')

I activated both interfaces, intending to start testing nftables, and set up the Test machine with a static address of 172.18.90.2/24 and gateway 172.18.90.1. I was only expecting to be able to ping 172.18.90.1 from the Test machine with this setup, but was extremely surprised to learn that I could not only ping any host, but further was able to run a web browser on the test machine with no problems connecting to anywhere. Somehow the Arch server was doing forwarding and address translation even though I hadn't installed iptables or nftables (yet).

So, it turns out that iptables is a dependency of netctl?

ibis:~iptables$ pactree -r iptables
iptables
..iproute2
  ..netctl

ibis:~iptables$ pacman -Q iptables
iptables 1.4.21-1

The first question is

Why does netctl need iptables?

I was quite shocked to see NAT happening without my having done anything to set it up.

But beyond this, the default firewall rules installed with the package look like this:

ibis:~iptables$ cat /etc/iptables/simple_firewall.rules
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p icmp -j ACCEPT 
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -j REJECT --reject-with tcp-reset 
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable 
-A INPUT -j REJECT --reject-with icmp-proto-unreachable 
COMMIT

There's no mention of the NAT table, so presumably there is some kind of default NAT behavior that is being invoked? This is my second question:

When you install iptables, NAT works by default somehow?

The final mystery is I never started the iptables service; in particular didn't even know that iptables had been installed as a dependency until NAT was magically working. When I check

systemctl | grep iptables

Nothing comes up. Third question:

Who/what/why is the iptables service being automatically activated, if indeed this is what is happening?

Edit: I'm an idiot. See my last comment on this thread for an explanation of why I thought ip forwarding was being enabled without my explicit permission.

Last edited by pgoetz (2014-06-12 10:07:43)

Leonid.I · 2014-06-10 15:35:25

1. Netctl needs iproute2, which in turn needs iptables; netctl doesn't use iptables directly.

2. Network packet processing (a.k.a. firewall) lives in the kernel. Iptables is just a frontend, so that you can talk to the kernel. The iptables systemd service (iptables.service) simply loads your ruleset into the kernel. Systemctl shows that iptables.service is not active, so why do you ask? When no ruleset is loaded, the kernel does not filter/drop anything, so if ip_forward is >0, your system will be a 'transparent' router. Moreover, if you _load_ ruleset which doesn't specify a table, like NAT, only defaults apply for that table (in the case of NAT they are all ACCEPT).

What does 'cat /proc/sys/net/ipv4/ip_forward' show? Do you have it setup via sysctl: 'grep ip_forward /etc/sysctl.d/* /usr/lib/sysctl.d/* /etc/sysctl.conf'? Also, you can print all your ip tables with 'for t in filter nat mangle; do echo $t; iptables -t $t -L -v; done'.

pgoetz · 2014-06-10 16:59:46

Leonid.I wrote:

1. Netctl needs iproute2, which in turn needs iptables; netctl doesn't use iptables directly.

Thanks. Yes I noticed that after I posted this question. That still doesn't explain why iproute2 depends on iptables. I emailed the package maintainer to try and find out why. In particular, iproute on my Ubuntu systems does not depend on iptables.

With nftables looking to replace iptables, this issue is going to come up even if there is a good reason for iproute2 to depend on iptables.

pgoetz · 2014-06-11 01:18:28

Leonid.I wrote:

What does 'cat /proc/sys/net/ipv4/ip_forward' show? Do you have it setup via sysctl: 'grep ip_forward /etc/sysctl.d/* /usr/lib/sysctl.d/* /etc/sysctl.conf'? Also, you can print all your ip tables with 'for t in filter nat mangle; do echo $t; iptables -t $t -L -v; done'.

Thanks, your comments were helpful. I'm still not sure why my system was (apparently) forwarding and NATing packets:

ibis:~~$ cat /proc/sys/net/ipv4/ip_forward
0
ibis:~~$ grep ip_forward /etc/sysctl.d/* /usr/lib/sysctl.d/* /etc/sysctl.conf
grep: /etc/sysctl.d/*: No such file or directory
grep: /etc/sysctl.conf: No such file or directory
ibis:~~$

pgoetz · 2014-06-11 04:02:38

Oh, I forgot to add that the iptables comes with some default rules that I don't quite understand:

ibis:~~$ cat /etc/iptables/simple_firewall.rules
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p icmp -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
COMMIT

Since I'm fairly new to systemd, I'm not sure there isn't some systemd magic which automatically invokes iptables when, for example, another machine is sending packets that should be forwarded to an interface.

fukawi2 · 2014-06-11 04:20:37

*filter

All the following commands are to be applied to the 'filter' table.

:INPUT DROP [0:0]

The default policy for the INPUT chain is DROP

:FORWARD DROP [0:0]

The default policy for the FORWARD chain is DROP

:OUTPUT ACCEPT [0:0]

The default policy for the OUTPUT chain is DROP

-A INPUT -p icmp -j ACCEPT

ACCEPT all incoming packets of protocol ICMP

-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

ACCEPT all incoming packets that belong to a RELATED or ESTABLISHED connection in the state tracking table.

-A INPUT -i lo -j ACCEPT

ACCEPT any packets that come "in" the loopback interface (lo).

-A INPUT -p tcp -j REJECT --reject-with tcp-reset

REJECT any incoming TCP packets by sending a TCP RST packet.

-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable

REJECT any incoming UDP packets by sending an ICMP port-unreachable packet.

-A INPUT -j REJECT --reject-with icmp-proto-unreachable

REJECT any other packets by sending an ICMP protocol-unreachable packet.

COMMIT

COMMIT all these rules (above) into the kernel (atomically unloading any existing rules first).

Remember that rules process in order, so TCP packets will get REJECT'ed by the third-last rule, and never get to the second last and last rules.

You've checked ip_forward now, but is the host still forwarding packets? The kernel will NEVER forward packets without ip_forward being != 0.

pgoetz · 2014-06-11 15:37:01

fukawi2 wrote:

You've checked ip_forward now, but is the host still forwarding packets? The kernel will NEVER forward packets without ip_forward being != 0.

Thanks, that was an awesome explanation. I've worked with iptables several times, but was unfamiliar with the *filter syntax -- where is this documented?

Yes, as far as I know the host is still forwarding packets, by which I mean if I attach the "interior" port to a switch and also attach a laptop with a statically configured IP address (as outlined in the initial post), the laptop can browse the Internet, (In particular, I didn't run tcpdump or wireshark to check out the packet flow directly.)

I haven't tried the experiment again since checking to make sure that /proc/sys/net/ipv4/ip_forward was set to 0, but nothing has changed on the machine (i.e. I haven't installed any new packages or changed any configurations), so presumably it's still happening. I can double check to confirm at the next opportunity

fukawi2 · 2014-06-11 23:00:08

pgoetz wrote:

unfamiliar with the *filter syntax -- where is this documented?

Probably documented somewhere......

I will be incredibly surprised if it is actually passing packets without ip_forward enabled. Post the output of the following commands from both the Arch server, and the test machine (laptop?):

ip a s
ip r s
mtr google.com

(You may need to install mtr if you don't already have it)

pgoetz · 2014-06-12 10:04:24

While I appreciate finding out about mtr -- this looks like an awesome network debugging tool, I'm very sorry to have wasted your time on this issue. The laptop I was using for this test is running Windows 7 (lessons learned: next time dig up a linux laptop before posting a problem like this). When I tried to repeat the experiment one more time just to be sure, this time the behavior was as expected: I could ping 172.18.90.1, but couldn't connect to any other host. The wireless router at my house is dual band, and Windows 7 has (I just noticed) the annoying habit of automatically connecting you to the second most popular available wireless network when you manually disconnect the one you're connected to (why someone thought this was sensible behavior is another matter). What must have happened is that I disconnected from the wireless network and then mindlessly forgot to check if the laptop had automatically reconnected on the other band. So pinging the statically connected port worked, and pinging google worked, too (using the wireless network I didn't realize I was connected to). I should know better than this, and owe you a beer. If you email me a method to buy you one from Texas, I will take care of that.

Leonid.I · 2014-06-12 19:00:23

Just for completeness: the complete iptables documentation is in the manpages: man 8 iptables{,-restore,-save,-extensions,-xml}. Also, some relevant but a bit outdated 'tutorials' can be found at http://cipherdyne.org/LinuxFirewalls/ .

Edit: That's just one site I used most; fukawi2 knows more of those...

HTH

Last edited by Leonid.I (2014-06-12 19:13:20)

pgoetz · 2014-06-12 20:03:58

Leonid.I wrote:

Also, some relevant but a bit outdated 'tutorials' can be found at http://cipherdyne.org/LinuxFirewalls/.

Thanks for the tip! I'm planning on using nftables, if only because I want to set up this server one time and then mostly forget about it, and nftables appears to be the wave of the future. Of course the nftables documentation is even worse than that for iptables. <:(

I found this linux home networking page to be the single best iptables resource when I was trying to update a somewhat complicated tables configuration someone else had set up.

fukawi2 · 2014-06-12 23:01:27

pgoetz wrote:

While I appreciate finding out about mtr -- this looks like an awesome network debugging tool,

It is

pgoetz wrote:

I'm very sorry to have wasted your time on this issue.

No problem

pgoetz wrote:

What must have happened is that I disconnected from the wireless network and then mindlessly forgot to check if the laptop had automatically reconnected on the other band.

I did suspect this was the case when I saw the Netgear WNDR3700 had Wireless

Arch Linux

#1 2014-06-09 18:25:02