Arch Linux

mike303

Member

Registered: 2016-03-14

Posts: 7

strange load balancing with ip-route over three WAN connection

hi folks,

i have some troubble with my setup:
eth0 is on borad nic
eth1-eth4 is a 4 port gigabit ethernet PCIe Card from Intel

eth0 -> 192.168.2.1 -> LAN
eth1 -> 192.168.101.2 -> 192.168.101.1 (ISP1 router)
eth2 -> 192.168.102.2 -> 192.168.102.1 (ISP2 router)
eth3 -> 192.168.103.2 -> 192.168.103.1 (ISP3 router)
eth4 not used

the internet connection on all 3 routers are ok and i can test it on the web interface on each router.

i setup the default route via:

root@idefix:/root# ip route add default scope global nexthop via 192.168.101.1 dev eth1 weight 1 nexthop via 192.168.102.1 dev eth2 weight 1 nexthop via 192.168.103.1 dev eth3 weight 1

root@idefix:/root# ip route show table all
default
        nexthop via 192.168.101.1  dev eth1 weight 1
        nexthop via 192.168.102.1  dev eth2 weight 1
        nexthop via 192.168.103.1  dev eth3 weight 1
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.1
192.168.101.0/24 dev eth1 proto kernel scope link src 192.168.101.2
192.168.102.0/24 dev eth2 proto kernel scope link src 192.168.102.2
192.168.103.0/24 dev eth3 proto kernel scope link src 192.168.103.2
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.2.0 dev eth0 table local proto kernel scope link src 192.168.2.1
local 192.168.2.1 dev eth0 table local proto kernel scope host src 192.168.2.1
broadcast 192.168.2.255 dev eth0 table local proto kernel scope link src 192.168.2.1
broadcast 192.168.101.0 dev eth1 table local proto kernel scope link src 192.168.101.2
local 192.168.101.2 dev eth1 table local proto kernel scope host src 192.168.101.2
broadcast 192.168.101.255 dev eth1 table local proto kernel scope link src 192.168.101.2
broadcast 192.168.102.0 dev eth2 table local proto kernel scope link src 192.168.102.2
local 192.168.102.2 dev eth2 table local proto kernel scope host src 192.168.102.2
broadcast 192.168.102.255 dev eth2 table local proto kernel scope link src 192.168.102.2
broadcast 192.168.103.0 dev eth3 table local proto kernel scope link src 192.168.103.2
local 192.168.103.2 dev eth3 table local proto kernel scope host src 192.168.103.2
broadcast 192.168.103.255 dev eth3 table local proto kernel scope link src 192.168.103.2
fe80::/64 dev eth2 proto kernel metric 256  pref medium
fe80::/64 dev eth1 proto kernel metric 256  pref medium
fe80::/64 dev eth0 proto kernel metric 256  pref medium
fe80::/64 dev eth3 proto kernel metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium
local ::1 dev lo table local proto none metric 0  pref medium
local fe80::21c:c4ff:fe47:ad88 dev lo table local proto none metric 0  pref medium
local fe80::21c:c4ff:fe47:ad89 dev lo table local proto none metric 0  pref medium
local fe80::21c:c4ff:fe47:ad8b dev lo table local proto none metric 0  pref medium
local fe80::3a2c:4aff:fee9:3655 dev lo table local proto none metric 0  pref medium
ff00::/8 dev eth2 table local metric 256  pref medium
ff00::/8 dev eth1 table local metric 256  pref medium
ff00::/8 dev eth0 table local metric 256  pref medium
ff00::/8 dev eth3 table local metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium

root@idefix:/root# cat /etc/iptables/iptables.rules
# Generated by iptables-save v1.6.0 on Sun Dec 18 09:16:49 2016
*nat
:PREROUTING ACCEPT [445:44418]
:INPUT ACCEPT [232:18010]
:OUTPUT ACCEPT [150:11223]
:POSTROUTING ACCEPT [144:10806]
-A POSTROUTING ! -d 192.168.2.0/24 -o eth1 -j SNAT --to-source 192.168.101.2
-A POSTROUTING ! -d 192.168.2.0/24 -o eth2 -j SNAT --to-source 192.168.102.2
-A POSTROUTING ! -d 192.168.2.0/24 -o eth3 -j SNAT --to-source 192.168.103.2
-A POSTROUTING -o eth1 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o eth3 -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [408:36550]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1935:319252]
-A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -o eth1 -j ACCEPT
-A FORWARD -i eth0 -o eth2 -j ACCEPT
-A FORWARD -i eth0 -o eth3 -j ACCEPT
COMMIT
# Completed on Sun Dec 18 09:16:49 2016

also /proc/sys/net/ipv4/ip_forward is 1

with this setup some connection run to nirvana.
actually all connection try over eth3!
tcpdump for all 3 external interfaces is running to look up whats happend
DNS is working
for example:

ping heise.de ok
IP 192.168.101.2 > 193.99.144.80: ICMP echo request, id 3308, seq 4, length 64
IP 193.99.144.80 > 192.168.101.2: ICMP echo reply, id 3308, seq 4, length 64

ping spiegel.de ok
IP 192.168.102.2 > 62.138.116.25: ICMP echo request, id 3328, seq 3, length 64
IP 62.138.116.25 > 192.168.102.2: ICMP echo reply, id 3328, seq 3, length 64

ping dresden.de gives 100% packet lost BUT on the outgoing interface (eth3) i see 'ECHO REQUEST' and 'ECHO REPLAY'
IP 192.168.103.2 > 194.49.19.111: ICMP echo request, id 3330, seq 2, length 64
IP 194.49.19.111 > 192.168.103.2: ICMP echo reply, id 3330, seq 2, length 64

Deleting the default route over eth3 help's but it's not a option

Has enyone a hint where the devil strikes

mike303

Member

Registered: 2016-03-14

Posts: 7

Re: strange load balancing with ip-route over three WAN connection

can be closed.

The error was on the router. I think the router interface fragmented the ip packet. I used a other interface on the router and all was fine....