[SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

robotanarchy · 2012-11-21 21:01:37

EDIT: my final script is in the post #16 with a short description of what it does!

Hello dear arch community,

I've been trying to set the following stuff up on my server:
- traffic shaping and policying (shape upload and download traffic) of all traffic that leaves the LAN on my arch box
- give SSH, web interfaces of programs like sabnzbd and WLAN users more priority

Some information about my archlinux box
- it is connected to the internet with a router inbetween
- it acts as wireless access point (that already works)
- it stores lots of files, that I want to access fast via LAN. but I don't want it to suck up all the traffic when it runs a download or when I access files on the go over the internet
- it has a vpn, that I only want to use over the LAN

I have tried out various tools (for example trickle - wrote my own archwiki article there), but all have failed me so far
After lots of lots of reading, I decided to try iptables.

I've used this guideas a base, figured out that it doesn't work that way (you can't make sub-classes of classes?), modified my approach, but I still can't get it to work.

Here's my script, that should set up all the iptables and tc stuff:

# config
device="br0"
lan_ip="192.168.2.131"

inet_burst=6k

inet_up=500kbit
inet_down=500kbit


port_sabnzbd=28093
port_pyload=31431
port_ssh=12934

# Classes:
#
#      1:  (qdisc) lan(11)
#      |
#      |----|
#      |    | (inet down, inet up)
#      |    |
#      |    |
#      |    1:3  inet_up(13) (Webinterfaces, SSH, WLAN, all other traffic)
#      |
#      |
#      1:2 inet_down(12)  (Webinterfaces, SSH, WLAN, all other traffic)
#

echo "setting up classes..."

tc qdisc add dev ${device} root handle 1: htb default 22

echo "1:2..." && tc class add dev ${device} parent 1:  classid 1:2 htb rate ${inet_down} ceil ${inet_down} burst ${inet_burst}
echo "1:3..." && tc class add dev ${device} parent 1:  classid 1:3 htb rate ${inet_up}   ceil ${inet_up}   burst ${inet_burst}



# put marks from IPTABLES to the right classes
echo "adding some iptables->classes glue with tc..."
tc filter add dev ${device} protocol ip parent 1: prio 3 handle 11 fw classid 1:
tc filter add dev ${device} protocol ip parent 1: prio 2 handle 12 fw classid 1:2
tc filter add dev ${device} protocol ip parent 1: prio 2 handle 13 fw classid 1:3

# high priorty inet mark
tc filter add dev ${device} protocol ip parent 1: prio 1 handle 22 fw classid 1:2
tc filter add dev ${device} protocol ip parent 1: prio 1 handle 23 fw classid 1:3



# add iptables rules
echo "adding iptables rules..."




# lan
iptables -I PREROUTING -p all -i ${device} -s "192.168.2.0/24" -d "192.168.2.0/24" -t mangle -j MARK --set-mark 11

# web interfaces
iptables -I PREROUTING -p TCP -i ${device} -s ${lan_ip} --sport ${port_sabnzbd} -t mangle -j MARK --set-mark 12 -v
iptables -I PREROUTING -p TCP -i ${device} -s ${lan_ip} --sport ${port_pyload}  -t mangle -j MARK --set-mark 12 -v
iptables -I PREROUTING -p TCP -i ${device} -d ${lan_ip} --dport ${port_sabnzbd} -t mangle -j MARK --set-mark 31 -v
iptables -I PREROUTING -p TCP -i ${device} -d ${lan_ip} --dport ${port_pyload}  -t mangle -j MARK --set-mark 31 -v


# ssh
iptables -I PREROUTING -p TCP -i ${device} -s ${lan_ip} --sport ${port_ssh}     -t mangle -j MARK --set-mark 22 -v
iptables -I PREROUTING -p TCP -i ${device} -d ${lan_ip} --dport ${port_ssh}     -t mangle -j MARK --set-mark 23 -v


# wlan + downloads + everything else
iptables -I PREROUTING -p all -i ${device} -s ${lan_ip} -t mangle -j MARK --set-mark 12 -v
iptables -I PREROUTING -p all -i ${device} -d ${lan_ip} -t mangle -j MARK --set-mark 13 -v

It doesn't give any errors anymore, but when I try to wget the newest ubuntu release for example, I still have the full speed (it should be in the 1:2 class and therefore limited!)

Oh and I am not even sure if $device should be eth0 or br0 (since I also use br0 for dhcpcd), but I've tried both over and over -- nothing works

So if anyone could help me with that, it would be greatly appreciated! You may also suggest me another tool if you know one that fits (although doing the hard way and learning what's wrong here would be preferred and more arch-way-ish of course )

Thanks a lot for reading!

Here's some more console output:

% iptables-save
% ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 6000 qdisc pfifo_fast master br0 state UP qlen 1000
    link/ether 00:01:2e:27:59:e0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::201:2eff:fe27:59e0/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP qlen 1000
    link/ether 00:24:23:08:dd:a0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::224:23ff:fe08:dda0/64 scope link
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 6000 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
14: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP
    link/ether 00:01:2e:27:59:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.131/24 brd 192.168.2.255 scope global br0
    inet6 fe80::201:2eff:fe27:59e0/64 scope link
       valid_lft forever preferred_lft forever


% ip route show table all
default via 192.168.2.1 dev br0  metric 214
10.8.0.0/24 via 10.8.0.2 dev tun0
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1
192.168.2.0/24 dev br0  proto kernel  scope link  src 192.168.2.131  metric 214
local 10.8.0.1 dev tun0  table local  proto kernel  scope host  src 10.8.0.1
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
broadcast 192.168.2.0 dev br0  table local  proto kernel  scope link  src 192.168.2.131
local 192.168.2.131 dev br0  table local  proto kernel  scope host  src 192.168.2.131
broadcast 192.168.2.255 dev br0  table local  proto kernel  scope link  src 192.168.2.131
local ::1 dev lo  proto kernel  metric 256
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev eth0  proto kernel  metric 256  mtu 6000
fe80::/64 dev wlan0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 hoplimit 255
local ::1 via :: dev lo  table local  proto none  metric 0  rtt 10ms rttvar 10ms cwnd 10
local fe80::201:2eff:fe27:59e0 via :: dev lo  table local  proto none  metric 0
local fe80::201:2eff:fe27:59e0 via :: dev lo  table local  proto none  metric 0
local fe80::224:23ff:fe08:dda0 via :: dev lo  table local  proto none  metric 0
unreachable ff00::/8 dev lo  table local  metric 256  error -101
ff00::/8 dev eth0  table local  metric 256  mtu 6000
ff00::/8 dev wlan0  table local  metric 256
ff00::/8 dev br0  table local  metric 256
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 hoplimit 255

EDIT: I forgot to add this:

% iptables -L -v
Chain INPUT (policy ACCEPT 4657 packets, 2120K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 7341 packets, 4496K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 3550 packets, 552K bytes)
 pkts bytes target     prot opt in     out     source               destination

Shouldn't there be some other rules listed or something?

Last edited by robotanarchy (2012-11-29 10:31:35)

fukawi2 · 2012-11-21 22:13:44

robotanarchy wrote:

EDIT: I forgot to add this:

% iptables -L -v
Chain INPUT (policy ACCEPT 4657 packets, 2120K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 7341 packets, 4496K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 3550 packets, 552K bytes)
 pkts bytes target     prot opt in     out     source               destination

Shouldn't there be some other rules listed or something?

Yes, but they should go in the mangle table:

iptables -nvL -t mangle

robotanarchy · 2012-11-25 21:19:25

Thanks, fukawi2!

Your post helped me a lot, because with

watch -n 1 iptables -nvL -t mangle

it became possible to find out what's going on. And that really kept me going

There was a lot wrong in my script, but I finally got the basic stuff to work (seperate speed for LAN and INET) and I think I understand it enough now (manpages tc and tc-htb help a lot) to add what I need

If anyone faces a similar problem, this (working!) script should get you started:

http://mldonkey.sourceforge.net/TrafficShaping

EDIT: not marking this thread as solved yet, as I might still have some questions

Last edited by robotanarchy (2012-11-25 21:58:22)

fukawi2 · 2012-11-25 22:03:11

No problems; I'm not that familiar with tc etc, but your iptables issue stood out

brebs · 2012-11-25 22:31:11

Here's my traffic-shaping setup.

robotanarchy · 2012-11-25 22:50:05

Currently I have the LAN up/down speed unlimited and the upload inet speed limited.

My problem: Limit inet download speed.

I have a big issue with this line:

iptables -I POSTROUTING -d ${lan_ip} -t mangle -j MARK --set-mark 13 # 0xd

When used like that, with POSTROUTING, the rule does not get any hits. It does get all the hits however, when I use PREROUTING or INPUT, but then "tc" ignores the marks.

I don't quite understand the iptables chain with prerouting, input, output, postrouting and where traffic shaper (tc) kicks in.

There are even some scripts out there that use prerouting (like the one I originally based mine on), but I can't get them to work.

Do I need to tell traffic shaper to look at the prerouting traffic somehow? Is this even possible?

Here's my full script:

# config
device="br0"
lan_ip="192.168.2.131"

inet_burst="6k"

inet_up="100000bit"
inet_down="100000bit"


# reset everything
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F
iptables -t nat -F
iptables -t mangle -F
tc qdisc del dev $device root 2> /dev/null
iptables -F POSTROUTING -t mangle



# Classes:
#
#      1:  (qdisc)
#      |
#      |----|----|
#      |    |    | (inet down, inet up)
#      |    |    |
#      |    |    |
#      |    |    1:13  inet_down(13) (Webinterfaces, SSH, WLAN, all other traffic)
#      |    |
#      |    |
#      |    1:12 inet_up(12)  (Webinterfaces, SSH, WLAN, all other traffic)
#      |
#      |
#      1:11 lan_traffic (11)

echo "setting up tc qdisc and classes..."

tc qdisc add dev ${device} root handle 1: htb default 13

echo "1:11..." && tc class add dev ${device} parent 1:  classid 11 htb rate 1tbps burst 12k
echo "1:12..." && tc class add dev ${device} parent 1:  classid 12 htb rate ${inet_up} burst ${inet_burst}
echo "1:13..." && tc class add dev ${device} parent 1:  classid 13 htb rate ${inet_down}   burst ${inet_burst}



# put marks from IPTABLES to the right classes
echo "adding tc filters to map iptables to classes..."
tc filter add dev ${device} parent 1:0 prio 3 protocol ip handle 11 fw flowid 1:11
tc filter add dev ${device} parent 1:0 prio 2 protocol ip handle 12 fw flowid 1:12
tc filter add dev ${device} parent 1:0 prio 2 protocol ip handle 13 fw flowid 1:13

# high priorty inet mark
#tc filter add dev ${device} protocol ip prio 1 handle 22 fw classid 12
#tc filter add dev ${device} protocol ip prio 1 handle 23 fw classid 13



echo "adding iptables rules..."

# POSTROUTING for upload traffic


echo "lan and vpn..." # needs to be the last rule, so put it on top here
iptables -I POSTROUTING -s "192.168.2.0/24" -d "192.168.2.0/24" -t mangle -j MARK --set-mark 11 # 0xb

echo "wlan, downloads, everything else..."
iptables -I POSTROUTING -s ${lan_ip} -t mangle -j MARK --set-mark 12 # 0xc


# PREROUTING for download traffic

#
# HELP
#
# only works with PREROUTING or INPUT but then tc doesn't seem to use the marks!
#
#
#
iptables -I PREROUTING -d ${lan_ip} -t mangle -j MARK --set-mark 13 # 0xd


# ssh, webinterfaces: todo, but that's not so hard

And thanks brebs, for posting your script!

Last edited by robotanarchy (2012-11-25 22:52:06)

brebs · 2012-11-25 22:56:05

robotanarchy wrote:

Limit inet download speed.

Who says that's a good idea?

AFAICT, that's pointless. Just shape the uploading, and let the downloading be done as normal (we cannot control the downloading).

Edit: Added link.

Last edited by brebs (2012-11-25 23:00:36)

robotanarchy · 2012-11-25 23:00:02

brebs wrote:

robotanarchy wrote:
Limit inet download speed.
Who says that's a good idea?
AFAICT, that's pointless. Just shape the uploading, and let the downloading be done as normal.

From what I have read, you can't directly control download speed, but at least you can set up a policy that ACKs get delayed/dropped and hopefully the host won't send that much data then.

I would like to have this, because the server is in a LAN environment and its downloads shouldn't block others from streaming youtube for example.

EDIT: If you know a better way, let me know. But I can not modify the hosts I download from to send less data

Last edited by robotanarchy (2012-11-25 23:00:49)

brebs · 2012-11-25 23:08:24

robotanarchy wrote:

ACKs get delayed/dropped

That would just cause *more* TCP re-synchronization/handshaking traffic. Counter-productive

robotanarchy · 2012-11-25 23:09:33

well it works in download managers I use what would you suggest then?

brebs · 2012-11-25 23:29:05

Download managers just add a crappy artificial speed limit. They aren't capable of using all the appropriately-allocated available bandwidth, as it dynamically ebbs and flows. This is why iptables shaping/sharing is better.

I've already said what to do. Concentrate on shaping uploads only.

Are you afraid of the LAN being saturated? You say "its downloads" - do you mean the server's downloads? I'm struggling to understand your logic.

robotanarchy · 2012-11-26 12:23:34

Oh sorry, what I meant with "the server's downloads" is when someone uploads something to the server.

Or when I want to download something with the server, in that case I mostly can't limit the speed on the other side.

Basic example:

I run

wget http://big/file

on the server to download something, that's where I would like to limit the speed

brebs · 2012-11-26 15:47:20

The speed is limited anyway, because the sender will be waiting for acknowledgement-of-receipt packets from your server.

And once the data has reached your server, it's gone beyond the appropriate time to drop it.

fukawi2 · 2012-11-26 22:21:31

robotanarchy wrote:

....set up a policy that ACKs get delayed/dropped

That IS controlling/policing the upload, not the download. There is no way to directly control how fast the other end sends data, and any indirect methods are inaccurate.

robotanarchy · 2012-11-27 22:30:39

brebs wrote:

The speed is limited anyway, because the sender will be waiting for acknowledgement-of-receipt packets from your server.
And once the data has reached your server, it's gone beyond the appropriate time to drop it.

As far as I know, it works like this:

Drop ACK packages that supply more packages than I want per second -> The server (probably) sends less data per second -> Less packages/data/traffic per second for real

fukawi2 wrote:

robotanarchy wrote:
....set up a policy that ACKs get delayed/dropped
That IS controlling/policing the upload, not the download. There is no way to directly control how fast the other end sends data, and any indirect methods are inaccurate.

Well, I prefer inaccurate indirect control over no control

I'm currently playing with "ingress", let's see if that gets me anywhere.

Last edited by robotanarchy (2012-11-27 22:30:54)

robotanarchy · 2012-11-28 16:54:18

Yay, it works now like I want it to

Here's my script if anyone wants something similar. I have read through mastershaper scripts, that helped a lot

again, here's the purpose:
Shape/Policy all traffic to a maximum down-/ and upload speed, except for LAN traffic (which is always fullspeed). Also SSH and (in this case pyload and sabnzbd)-webinterfaces are prioritized, customize to your needs

device="br0"
lan_ip="192.168.2.131"
lan_mask="192.168.2.0/24"
inet_burst="10k"
inet_up="1000kbps"
inet_down="3000kbps"
port_sabnzbd=3233
port_pyload=5334
port_ssh=1123


echo "reseting everything traffic shaping related..."
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F
iptables -t nat -F
iptables -t mangle -F
tc qdisc del dev $device root 2> /dev/null
tc qdisc del dev $device ingress 2> /dev/null
iptables -F POSTROUTING -t mangle


echo "setting up tc qdiscs..."
echo "  ingress..."								&& tc qdisc add dev ${device} handle ffff: ingress
echo "  htb..."									&& tc qdisc add dev ${device} root handle 1: htb default 13
echo "seting up classes..."
echo "  1:11..."								&& tc class add dev ${device} parent 1:  classid 11 htb rate 1tbps burst 12k
echo "  1:22..."								&& tc class add dev ${device} parent 1:  classid 12 htb rate ${inet_up} burst ${inet_burst}
echo "adding tc filters to map marks from iptable rules to classes..."
echo "  filter 11(0x0b)->1:11 (prio 3)..."		&& tc filter add dev ${device} parent 1:0 prio 3 protocol ip handle 11 fw flowid 1:11
echo "  filter 12(0x0c)->1:12 (prio 2)..."		&& tc filter add dev ${device} parent 1:0 prio 2 protocol ip handle 12 fw flowid 1:12
echo "  filter 22(0x16)->1:12 (prio 1)..."		&& tc filter add dev ${device} parent 1:0 protocol ip prio 1 handle 22 fw flowid 1:12
echo "adding iptables rules..."
echo "  lan and vpn..."							&& iptables -I POSTROUTING 			-s ${lan_mask}	-d ${lan_mask}			-t mangle -j MARK --set-mark 11 # 0xb
echo "  ssh..."									&& iptables -I POSTROUTING -p TCP	-s ${lan_ip}	--sport ${port_ssh}		-t mangle -j MARK --set-mark 22 # 0x16
echo "  sabnzbd..."								&& iptables -I POSTROUTING -p TCP	-s ${lan_ip}	--sport ${port_sabnzbd}	-t mangle -j MARK --set-mark 22 # 0x16
echo "  pyload..."								&& iptables -I POSTROUTING -p TCP	-s ${lan_ip}	--sport ${port_pyload}	-t mangle -j MARK --set-mark 22 # 0x16
echo "  wlan, downloads, everything else..."	&& iptables -I POSTROUTING			-s ${lan_ip}							-t mangle -j MARK --set-mark 12 # 0xc
echo "adding download limit filter..."
echo "  lan..."									&& tc filter add dev ${device} parent ffff: protocol ip prio 1 u32 match ip src ${lan_mask} flowid :1
echo "  other..."								&& tc filter add dev ${device} parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate ${inet_down} burst ${inet_burst} drop flowid :1

EDIT: modified the "lan..." line according to this thread: http://forums.opensuse.org/english/get- … luded.html because it didn't really do what I wanted before

Last edited by robotanarchy (2012-12-11 23:40:33)

brebs · 2012-11-28 17:23:12

What benefit do you see from those two "police rate" lines? AFAICT, they only make sense if there's some other traffic on your LAN (e.g. PC-to-PC, not going through the server) which you don't want to congest.

Which is a different reason to the ones you were giving

robotanarchy · 2012-11-29 10:28:29

brebs wrote:

What benefit do you see from those two "police rate" lines? AFAICT, they only make sense if there's some other traffic on your LAN (e.g. PC-to-PC, not going through the server) which you don't want to congest.
Which is a different reason to the ones you were giving

Oh that's the point where you didn't understand me? Because that was exactly the point. There are lots of PCs on the LAN, which directly get stuff from the internet, not through the pc where the script runs on. Like this:

router ----> archbox_with_that_script ---> only a few mobiles/pcs with wlan
       '------> many more pcs

I hope it's clear now and you're right, I've read my posts again and I didn't make that clear earlier, sorry for that

Arch Linux

#1 2012-11-21 21:01:37

[SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#2 2012-11-21 22:13:44

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#3 2012-11-25 21:19:25

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#4 2012-11-25 22:03:11

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#5 2012-11-25 22:31:11

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#6 2012-11-25 22:50:05

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#7 2012-11-25 22:56:05

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#8 2012-11-25 23:00:02

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#9 2012-11-25 23:08:24

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#10 2012-11-25 23:09:33

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#11 2012-11-25 23:29:05

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#12 2012-11-26 12:23:34

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#13 2012-11-26 15:47:20

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#14 2012-11-26 22:21:31

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#15 2012-11-27 22:30:39

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#16 2012-11-28 16:54:18

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#17 2012-11-28 17:23:12

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

#18 2012-11-29 10:28:29

Re: [SOLVED]Traffic shaping with iptables+tc, but only for non-lan traffic

Board footer