You are not logged in.

#1 2011-08-02 21:49:07

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

[SOLVED] iptables load balancing help - nth mode

I was wondering if someone can help me figure out what is wrong with my rules, I am completely stuck.
I am trying to create a load balancer host (192.168.32.67) that will load balance DNS queries (port 53 tcp and udp) round robin fashion to 192.168.2.197 and 192.168.2.252 via nth mode from the statistic module. The problem is that when I try and do nth balancing every 1 to 4 queries gets 'stuck', as in dig hangs, I see nothing logged in iptables and iptraf mentions something about ICMP destn port unreachable which doesn't make much sense. Below are my rules:

# enable forwarding
echo 1 >| /proc/sys/net/ipv4/ip_forward
# clear rules
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT

# marks for restoring existing connections
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t mangle -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info

# snat
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j LOG --log-prefix 'snat-source-192.168.32.67: ' --log-level info
iptables -t nat -A SNAT1 -p all -j SNAT --to-source 192.168.32.67

# dnats
iptables -t nat -N DNAT1
iptables -t nat -A DNAT1 -j LOG --log-prefix 'dnat-to-192.168.2.197: ' --log-level info
iptables -t nat -A DNAT1 -p udp --dport 53 -j DNAT --to-destination 192.168.2.197:53
iptables -t nat -A DNAT1 -p tcp --dport 53 -j DNAT --to-destination 192.168.2.197:53
iptables -t nat -A DNAT1 -j MARK --set-mark 1
iptables -t nat -A DNAT1 -j CONNMARK --save-mark

iptables -t nat -N DNAT2
iptables -t nat -A DNAT2 -j LOG --log-prefix 'dnat-to-192.168.2.252: ' --log-level info
iptables -t nat -A DNAT2 -p udp --dport 53 -j DNAT --to-destination 192.168.2.252:53
iptables -t nat -A DNAT2 -p tcp --dport 53 -j DNAT --to-destination 192.168.2.252:53
iptables -t nat -A DNAT2 -j MARK --set-mark 2
iptables -t nat -A DNAT2 -j CONNMARK --save-mark

# restore existing connections
iptables -t mangle -A PREROUTING -p udp --dport 53 -m state --state ESTABLISHED,RELATED -j RESTOREMARK
iptables -t mangle -A PREROUTING -p tcp --dport 53 -m state --state ESTABLISHED,RELATED -j RESTOREMARK

# round robin balance DNAT requests
iptables -t nat -A PREROUTING -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j DNAT1
iptables -t nat -A PREROUTING -m state --state NEW -m statistic --mode nth --every 2 --packet 1 -j DNAT2

# allow DNATS back through
iptables -t nat -A POSTROUTING -j SNAT1

Last edited by T-Dawg (2011-08-03 11:21:33)

Offline

#2 2011-08-02 23:58:32

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: [SOLVED] iptables load balancing help - nth mode

Try jumping RESTOREMARK on all traffic; if there isn't one then it will just set the mark to 0. Maybe it's getting confused there only restoring the mark for RELATED and ESTABLISHED.

Also, I'm confused with your SNAT; it SNAT's to 192.168.32.67 but you haven't said where that fits in to the network?

EDIT: Also, a tcpdump on the inside should reveal what's going on (perhaps not why, but at least what wink)

Last edited by fukawi2 (2011-08-02 23:59:15)

Offline

#3 2011-08-03 00:06:03

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: [SOLVED] iptables load balancing help - nth mode

Sorry, it was a posting error on the ip: s/10.200.0.21/192.168.32.67

I'm not sure what you mean by jumping RESTOREMARK on all traffic? If you mean also NEW then wouldn't that destroy my DNAT rules?

I tried tcpdump and all it showed was a connection to the load balancer host and that was it.

Offline

#4 2011-08-03 03:05:59

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: [SOLVED] iptables load balancing help - nth mode

This is what I would try...

# enable forwarding
echo 1 >| /proc/sys/net/ipv4/ip_forward
# clear rules
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT

# snat
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j LOG --log-prefix 'snat-source-192.168.32.67: ' --log-level info
iptables -t nat -A SNAT1 -p all -j SNAT --to-source 192.168.32.67

# DNAT outbound traffic
iptables -t nat -A POSTROUTING -j SNAT1

# dnats
iptables -t nat -N DNAT1
iptables -t nat -A DNAT1 -j LOG --log-prefix 'dnat-to-192.168.2.197: ' --log-level info
iptables -t nat -A DNAT1 -p udp --dport 53 -j DNAT --to-destination 192.168.2.197:53
iptables -t nat -A DNAT1 -p tcp --dport 53 -j DNAT --to-destination 192.168.2.197:53
iptables -t nat -A DNAT1 -j MARK --set-mark 1
iptables -t nat -A DNAT1 -j CONNMARK --save-mark

iptables -t nat -N DNAT2
iptables -t nat -A DNAT2 -j LOG --log-prefix 'dnat-to-192.168.2.252: ' --log-level info
iptables -t nat -A DNAT2 -p udp --dport 53 -j DNAT --to-destination 192.168.2.252:53
iptables -t nat -A DNAT2 -p tcp --dport 53 -j DNAT --to-destination 192.168.2.252:53
iptables -t nat -A DNAT2 -j MARK --set-mark 2
iptables -t nat -A DNAT2 -j CONNMARK --save-mark

# marks for restoring existing connections
iptables -t nat -N RESTOREMARK
iptables -t nat -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t nat -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info
iptables -t nat -A RESTOREMARK -m connmark --mark ! 0 -j ACCEPT

# round robin balance DNAT requests
iptables -t nat -N ROUNDROBIN
iptables -t nat -A ROUNDROBIN -m statistic --mode nth --every 2 --packet 0 -j DNAT1
iptables -t nat -A ROUNDROBIN -m statistic --mode nth --every 2 --packet 1 -j DNAT2

# restore and ACCEPT existing connections so they never touch the round-robin NATs
iptables -t mangle -A PREROUTING -j RESTOREMARK

# despatch NEW connections to the round-robin chain
iptables -t nat -A PREROUTING -m state --state NEW -p udp --dport 53 -j ROUNDROBIN
iptables -t nat -A PREROUTING -m state --state NEW -p tcp --dport 53 -j ROUNDROBIN

Offline

#5 2011-08-03 04:09:50

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: [SOLVED] iptables load balancing help - nth mode

Thanks fukawi but I figured out what the problem was. In the old days before nth was part of the statistics module --every 2 --packet 0....--every 2 --packet 1 would have been correct. Now there is no global counter and it is reset per rule. So, I needed to do --every 2 --packet 0.... --every 1 --packet 0 instead. Now it works perfectly. I'm still unclear whether the connection marking was really necessary (it worked either way) other than for the restoremark policy, will have to look at that closer tomorrow.

Thanks again for your help.

  Tyler

Offline

#6 2011-08-03 06:30:53

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: [SOLVED] iptables load balancing help - nth mode

T-Dawg wrote:

Now there is no global counter and it is reset per rule.

Errr, wut? That seems to defeat the purpose :-|
Surely that has to be some kind of regression in netfilter.

Offline

Board footer

Powered by FluxBB