You are not logged in.
Hello,
I'm having difficulties with properly configuring my multiple-ISP gateway. If I use IP rules based on tables only, it works. Here's the overview:
[root@router-4g ~]$ ip rule
0: from all lookup local
32765: from 192.168.1.0/24 lookup alt
32766: from all lookup main
32767: from all lookup default
[root@router-4g ~]$ ip route show table alt
default via 95.40.74.153 dev wwp0s3f2u4i2
95.40.74.152/29 dev wwp0s3f2u4i2 proto kernel scope link src 95.40.74.156
192.168.0.0/24 dev br0 proto kernel scope link src 192.168.0.10
192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.10
[root@router-4g ~]$ ip route
default via 5.185.99.69 dev wwp0s3f2u2i7
5.185.99.0/24 dev wwp0s3f2u2i7 proto kernel scope link src 5.185.99.69
95.40.74.152/29 dev wwp0s3f2u4i2 proto kernel scope link src 95.40.74.156 metric 204
192.168.0.0/24 dev br0 proto kernel scope link src 192.168.0.10
192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.10The problem arises if I use iptables to fwmark packets:
[root@router-4g ~]$ ip rule add fwmark 1 table alt
[root@router-4g ~]$ iptables -t mangle -A PREROUTING -p tcp --dport 80 -s 192.168.0.0/24 -j MARK --set-mark 1After using wireshark I noticed the packets find their way out (properly routed br0 -> wwp0s3f2u4i2), but for some reason when they get back, they never reach the original sender behind NAT...
Last edited by madman_xxx (2016-06-16 23:33:10)
Offline
The problem arises if I use iptables to fwmark packets:
[root@router-4g ~]$ ip rule add fwmark 1 table alt [root@router-4g ~]$ iptables -t mangle -A PREROUTING -p tcp --dport 80 -s 192.168.0.0/24 -j MARK --set-mark 1
Without seeing iptables rules it is hard to say, but if that is the only change you made then packets from the 192.168.0.0/24 may still be SNAT'ed with source address 5.185.99.69 like they presumable were before, even though they are now routed out the out via 95.40.74.153. If this is the case then the reply packets are likely dropped by the rp_filter.
Offline
Without seeing iptables rules it is hard to say, but if that is the only change you made then packets from the 192.168.0.0/24 may still be SNAT'ed with source address 5.185.99.69 like they presumable were before, even though they are now routed out the out via 95.40.74.153. If this is the case then the reply packets are likely dropped by the rp_filter.
If that's the case, what can be done? How to make proper masquerading? Also - why the packets are routed properly when only the source policy routing is used?
My (stripped) iptables config looks like this:
[root@router-4g ~]# iptables-save
*mangle
:PREROUTING ACCEPT [2673381:1818705094]
:INPUT ACCEPT [739207:474481338]
:FORWARD ACCEPT [1928777:1343762481]
:OUTPUT ACCEPT [687812:521084942]
:POSTROUTING ACCEPT [2616535:1864844687]
COMMIT
*filter
:INPUT DROP [3783:537189]
:FORWARD ACCEPT [1928723:1343759745]
:OUTPUT ACCEPT [687821:521086134]
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
...
-A INPUT -p gre -j ACCEPT
-A FORWARD -i br1 -o br0 -j DROP
COMMIT
*nat
:PREROUTING ACCEPT [30515:3345624]
:INPUT ACCEPT [5405:347188]
:OUTPUT ACCEPT [11367:999491]
:POSTROUTING ACCEPT [711:225309]
-A POSTROUTING -o wwp0s3f2u2i7 -j MASQUERADE
-A POSTROUTING -o wwp0s3f2u4i2 -j MASQUERADE
COMMITOffline
You were (kind of) right, branch. rp_filter messes with forwarding the reply. This is what helped:
sudo sysctl -w net.ipv4.conf.wwp0s3f2u4i2.rp_filter=2Not sure why this helps though, as the packets travel both ways via the wwp0s3f2u4i2... Also, meddling with rp_filter is not considered a solution as some people mention...
Offline
Not sure why this helps though, as the packets travel both ways via the wwp0s3f2u4i2... Also, meddling with rp_filter is not considered a solution as some people mention...
I presume it was set to 1 (strict mode) before? In strict mode, an incoming packet is rejected if a reply would not be routed to the interface the packet came in on. This check is not aware of any marks that may be set on an actual reply packet.
Example:
A) Packet 1 arrives on br0 with src_ip=192.168.0.5, dest_ip=123.45.67.89. It gets marked by iptables to be routed with the "alt" table which sends it to wwp0s3f2u4i2 where it gets SNAT'ed to src_ip=95.40.74.156.
B) Packet 2 (a reply to Packet 1) arrives on wwp0s3f2u4i2 with src_ip=123.45.67.89, dest_ip=95.40.74.156. The NAT code recognizes this as a reply to Packet 1 and sets dest_ip=192.168.0.5.
C) The routing table is consulted for a hypothetical packet with src_ip=192.168.0.5 and dest_ip=123.45.67.89. Since this hypothetical packet did not traverse iptables and was therefore not marked, the route is looked up in the "main" table which says it should go out wwp0s3f2u2i7. Since Packet 2 arrived on wwp0s3f2u4i2 but the reverse route check said a reply would go out wwp0s3f2u2i7, Packet 2 is dropped.
With rp_filter=2 (sloppy mode), the above packet is accepted because it would have been routed out an interface; it does not consider which interface.
If you want more control over how the rp_filter check is done you can set rp_filter=0 in sysctl and use the rpfilter iptables exension (see man iptables-extensions)
Offline