You are not logged in.

Pages: 1

#1 2022-07-11 14:39:51

sunnylx
Member
Registered: 2015-08-20
Posts: 43

[SOLVED] iptables masquerading not working on Arch

Let me see if I can more or less explain this as clearly as possible:

As a first note, all the following steps work fine on the RPI with kernel 5.10.63 with iptables 1.8.2 nf_tables version.

The same configuration does not work on a PC running Arch with kernel 5.18.10 with iptables 1.8.8 nf_tables version.

So I'm doing some tests with tayga (http://www.litech.org/tayga/) wich is a IPV6 NAT64 program to NAT Ipv6 addresses into IPv4 addresses.

Internally tayga has a pool of addresses that is used to do the nat64 translation.
This pool of addresses is normally a not used address range on the local network. So if my local network is 192.168.1.0/24, then the pool is 192.168.255.0/24 for example.

So if a packet from ipv6_1 arrives at tayga, it goes out as having a source ipv4_1 address (ex: 192.168.255.18) to the ipv4 destination, and a packet arriving from ipv6_2 goes out as having ipv4_2 as the source and so on.

This works by creating a network interface (normally named nat64), and adding a route to it for the ipv6 prefix, in my case 64:ff9b::/96 on the machine where the nat is running. Then tayga "grabs" to this interface and does the translation.

So if I want to ping google dns from IPv6, i can do: ping 64:ff9b::808:808 or ping 64:ff9b:8.8.8.8 and the packet goes to the nat64 interface (due to the route added previously), tayga transforms it into a packet with a src ip from the pool to the destination and sends it to the destination.

So basically:

Machine1: (ffaa:aa:1::23 - 64:ff9b::808:808) > route to machine tayga > route to interface nat64 -> Tayga -> extracts 808:808 and converts to 8.8.8.8 -> outs from nat64 interface as pool address  src to the destination ( example:  SRC: 192.168.255.18, DST:8.8.8.8).

This works fine in both ARCH and the RPI.

The issue is that the SRC (the 192.168.255.0/24) address is not in the range of the local network addresses. This could be solved by adding to the default router a route for the 192.168.255.0/24 with the tayga machine as the next hop, but in my case I can't add routes to my domestic provider router.

Which leaves to do NAT to the ips 192.168.255.0/24 to the local address space.

On the RPI, adding the following iptables rule:

iptables -t nat -A POSTROUTING -s 192.168.255.0/24 -j MASQUERADE

does the trick.

Tayga transforms the packet, and iptables nat rewrites the source address, and everything works as expected. I can ping goggle dns from ipv6 through NAT64 and again NAT44.

The problem is that on ARCH it doesn't work. The POSTROUTING rule is never called, and the packets always leave the nat64 interface to the eth0 (for example) interface without NAT44 (from the 192.168.255.X to 192.168.1.Y).

I've upgraded the iptables, to iptables-nft, reset all the rules, fluxed tables, reboots, added logging and so forth, nothing works on ARCH, it worked first  time on the RPI.

So:

iptables -t nat -L 

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            
MASQUERADE  all  --  192.168.255.0/24     anywhere            
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:tungsten-https
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:etlservicemgr
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:cslistener
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:irdmi

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp dpt:tungsten-https to:172.17.0.2:9443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:etlservicemgr to:172.17.0.3:9001
DNAT       tcp  --  anywhere             anywhere             tcp dpt:cslistener to:172.17.0.2:9000
DNAT       tcp  --  anywhere             anywhere             tcp dpt:irdmi to:172.17.0.2:8000
iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:tungsten-https
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:etlservicemgr
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:cslistener
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:irdmi

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Any ideas why the behavior is different on Arch when the same configuration is used on another machine and it works fine?

Last edited by sunnylx (2022-07-13 13:54:41)

Offline

#2 2022-07-11 22:18:34

seth
Member
Registered: 2012-09-03
Posts: 60,805

Re: [SOLVED] iptables masquerading not working on Arch

How many docker instances do you run on the raspi?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#3 2022-07-12 07:17:57

sunnylx
Member
Registered: 2015-08-20
Posts: 43

Re: [SOLVED] iptables masquerading not working on Arch

On the RPI, only one (Portainer) since I use it only for testing. Anyway both the RPI and the Arch machine both have Docker installed, but I've ruled out docker iptables rules as the reason since I've tested also with clean iptables.

Also configuration such as /proc/sys/net/ipv4/conf/all/forwarding = 1 and /proc/sys/net/ipv6/conf/all/forwardind = 1 is set in both machines.

The issue is that exactly the same configuration on the RPI works and on the Arch machine the only thing that fails is the IPv4 masquerading from the IPv6 -> IPv4 transformation to the local network addressing fails.

Adding iptables logging rules for the nat table, shows that for the INPUT chain, the chain is called, but the POSTROUTING chain is never called...

Last edited by sunnylx (2022-07-12 07:18:38)

Offline

#4 2022-07-12 09:20:41

-thc
Member
Registered: 2017-03-15
Posts: 775

Re: [SOLVED] iptables masquerading not working on Arch

sunnylx wrote:

Adding iptables logging rules for the nat table, shows that for the INPUT chain, the chain is called, but the POSTROUTING chain is never called...

This means the TCP/IP-Stack handles it like traffic for a local destination.

From a serverfault thread:

nat/INPUT will happen after the initial routing decision, only if traffic is deemed local

Offline

#5 2022-07-12 12:47:36

sunnylx
Member
Registered: 2015-08-20
Posts: 43

Re: [SOLVED] iptables masquerading not working on Arch

That might make sense but the destination ip address is not local.

For example, this sample packet ( 192.168.255.18  -> 8.8.8.8) comes out of the NAT64 interface and goes out to the eth0 interface is "natted" to (192.168.1.20 -> 8.8.8.8) on the RPI.
On arch it doesn't.

Network configuration on the PI:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:4c:93:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fd11:1111:1122:2222:e9ef:15e3:36d9:42f0/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 1792sec preferred_lft 1792sec
    inet6 fe80::3661:68d5:e9de:b7d3/64 scope link 
       valid_lft forever preferred_lft forever
....
....
9: nat64: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/none 
    inet 192.168.0.1/32 scope global nat64
       valid_lft forever preferred_lft forever
    inet6 fdaa:bb:1::1/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::118f:71a5:5124:33ff/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

On Arch:

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 9c:5c:8e:42:1b:f1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global noprefixroute enp2s0
       valid_lft forever preferred_lft forever
    inet6 fd11:1111:1122:2222:bcfb:e77f:a3b6:bf03/64 scope global dynamic noprefixroute 
       valid_lft 1777sec preferred_lft 1777sec
    inet6 fe80::476a:b3b5:db0f:344f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

9: nat64: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 500
    link/none 
    inet 192.168.0.1/32 scope global nat64
       valid_lft forever preferred_lft forever
    inet6 fdaa:bb:1::1/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::bd50:7d1e:e31a:cf63/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

On the RPI the iptables nat rules:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  192.168.255.0/24     0.0.0.0/0           
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:9001
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:9443
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:8000

And on Arch:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  192.168.255.0/24     0.0.0.0/0           
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:9443
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:9001
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:9000
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:8000

So the question remains why the POSTROUTING rule is processed on the RPI and not on ARCH...

Offline

#6 2022-07-12 16:01:43

-thc
Member
Registered: 2017-03-15
Posts: 775

Re: [SOLVED] iptables masquerading not working on Arch

Capture those packets via iptables log target ("-j LOG") - is the packet destination local?

Offline

#7 2022-07-12 20:21:01

seth
Member
Registered: 2012-09-03
Posts: 60,805

Re: [SOLVED] iptables masquerading not working on Arch

If the iptables look identical on both systems (minus the IPs), check the actual nftables.
Esp. firewalld puts stuff there and it won't show up in iptables, https://bbs.archlinux.org/viewtopic.php?id=271565

Edit: and also the defaults are rather restrictive and you should™ have nftables through iptables-nft, so look there.

Last edited by seth (2022-07-12 20:23:53)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#8 2022-07-13 10:54:58

sunnylx
Member
Registered: 2015-08-20
Posts: 43

Re: [SOLVED] iptables masquerading not working on Arch

-thc wrote:

Capture those packets via iptables log target ("-j LOG") - is the packet destination local?

The packet destination is 8.8.8.8, so definitely not local.

Now the interesting thing is that I can see the packet on the NAT64 interface (using wireshark) and not exiting on eth0 interface. So the issue might not be NAT but forwarding.

Despite this both net.ipv4.ip_forwarding and net.ipv6.conf.all.forwarding is both 1...  It is set on both the all and the interfaces proc instances.

Anyway, after adding the following rule as the *first* rule, the NAT started working albeit nothing more:

iptables -I POSTROUTING 1 -j ACCEPT

Now using wireshark I can see the packet leaving on eth0 with the NATted address ( from 192.168.255.X -> Eth0 IP) to 8.8.8.8, the ping reply, but not the ping reply from the eth0 to NAT64 interface to close the loop.

Offline

#9 2022-07-13 11:00:29

sunnylx
Member
Registered: 2015-08-20
Posts: 43

Re: [SOLVED] iptables masquerading not working on Arch

seth wrote:

If the iptables look identical on both systems (minus the IPs), check the actual nftables.
Esp. firewalld puts stuff there and it won't show up in iptables, https://bbs.archlinux.org/viewtopic.php?id=271565

Edit: and also the defaults are rather restrictive and you should™ have nftables through iptables-nft, so look there.

I got two machine with ARCH, one with iptables and nft after I upgraded when I started to debug this issue, and another without nft tables.

Both show the same behavior.  No firewalls or any filtering is installed/configured.

The issue is definitely with the iptables related, but not directly associated with nft tables.

As my above post, I keep debugging and adding a first rule an ACCEPT rule on the FORWARD chain, makes NAT work from NAT64 to ETH0, but the inverse path is still not working.

So right now I can get the ping reply to 8.8.8.8 on eth0 interface, but it is not forwarded to NAT64. Probably because the system thinks it does not need to forward it again...

Offline

#10 2022-07-13 13:53:50

sunnylx
Member
Registered: 2015-08-20
Posts: 43

Re: [SOLVED] iptables masquerading not working on Arch

Ok, more or less solved.

It was related to docker, but I'll have to find out why....

So basically I've disabled docker:

systemctl stop docker.service
systemctl stop docker.socket
systemctl disable docker.service
systemctl disable docker.socket

and rebooted.

Iptables (without nft_tables) came out clean (no rules at all for the filter an nat tables).
Also the Forward chain policy changed from policy DROP to policy ACCEPT

After reboot, verified that the net.ipv4.ip_forwarding was 1 and also ipv6.conf.all.forward was 1, added the iptables MASQUERADE rule, and it worked just fine.

So definitely the issue is related/associated with docker iptables rules.

Edit:  From the Arch Linux wiki: https://wiki.archlinux.org/title/Intern … forwarding (Check Troubleshooting)

This Github issue, is exactly the same problem: https://github.com/moby/moby/issues/43719

Last edited by sunnylx (2022-07-13 14:21:38)

Offline

Pages: 1

Board footer

Powered by FluxBB