Script to get bandwidth usage

Lazze · 2010-02-22 16:15:13

Hello

I'm currently looking for a shell, python, perl or any kind of simple application to monitor the bandwidth usage on my server. All I want it to do is to check once every second or something and get an average usage over the last five minutes. I'm going to use it within a php web application so when calling the application i would lite to get a easy to parse response.

If you have any ideas of such applications please let me know.

Thanks
// Lasse

Sin.citadel · 2010-02-22 16:49:59

do you want the entire system based usage, or is it ip-address specific, in either case, create an iptables rule, since the iptables firewall stores bytes for a specific rule, you can then parse it with bash and then store it somewhere for access with a web application.

Lazze · 2010-02-22 19:15:47

Sounds like a plan, I'll look into it and return with the results when I got something. Thanks!

n0dix · 2010-02-22 19:22:32

Try with this package: nethogs.
http://aur.archlinux.org/packages.php?ID=4526

johan84 · 2010-02-22 21:59:41

Sin.citadel wrote:

do you want the entire system based usage, or is it ip-address specific, in either case, create an iptables rule, since the iptables firewall stores bytes for a specific rule, you can then parse it with bash and then store it somewhere for access with a web application.

I want to try something similar: keep statistics of my internet traffic.
(Internet traffic and LAN pass through the same network card)

At the moment I use conky to see ul/dl-speed / total bytes transferred (but it shows the sum of LAN+www)

I tried some apps (like iptraf, downloadd, etc.) but I don't get the result I want.
I want to see total received/sent MB for LAN-only and internet-only. (if possible on an hourly/daily/monthly/yearly basis)

Here's a guide (http://www.linux.com/archive/feed/50649) for iptables I was following.
For some reason the FORWARD chain doesn't accept any packets (I attached my output).
The IP of my computer is 192.168.0.3.
So I figured (using the Chain www rules):
* rule 6 - rule 3 = internet download
* rule 5 - rule 2 = internet upload
* rule 1 - rule 4 = total internet traffic

The LAN addresses are 192.168.0.x

Is this the correct use for the iptables, or is there another way I'm supposed to do this? (or an application that does this for you)

If I want hourly/daily stats, I'll have to reset these stats manually and insert the values in a database or something?

[root@...]# iptables -L -v -n --line-numbers
Chain INPUT (policy ACCEPT 261K packets, 387M bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     261K  387M www        all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2     258K  383M lan        all  --  *      *       192.168.0.0/24       192.168.0.0/24      

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 lan        all  --  *      *       0.0.0.0/0            192.168.0.0/24      
2        0     0 lan        all  --  *      *       192.168.0.0/24       0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 161K packets, 8826K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     159K 8733K lan        all  --  *      *       0.0.0.0/0            192.168.0.0/24      
2     161K 8826K www        all  --  *      *       0.0.0.0/0            0.0.0.0/0           
3     159K 8733K lan        all  --  *      *       192.168.0.0/24       192.168.0.0/24      

Chain lan (5 references)
num   pkts bytes target     prot opt in     out     source               destination         
1     576K  400M            all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain www (2 references)
num   pkts bytes target     prot opt in     out     source               destination         
1     422K  396M            all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2     159K 8733K            all  --  *      *       192.168.0.3          192.168.0.0/24      
3     258K  383M            all  --  *      *       192.168.0.0/24       192.168.0.3         
4     417K  392M            all  --  *      *       192.168.0.0/24       192.168.0.0/24      
5     161K 8826K            all  --  *      *       192.168.0.3          0.0.0.0/0           
6     261K  387M            all  --  *      *       0.0.0.0/0            192.168.0.3

Sin.citadel · 2010-02-23 04:32:45

yes, it is right. but you can do any combination of rules to get the proper result, i myself have done this for an ip-specific rule so i can get the up/down usage for each of my network machines

also, you should use the -x parameter with iptables so all values are in bytes instead of MB/GB/KB, this makes it easier so you can simply add the values instead of converting them and then adding. Afterword you can use the -Z parameter with your specific chain to reset all the bytes for the rules.

i have a script for parsing with bash, and a python script which takes those values and puts them in a mysql database. if anyone needs it, i will upload it here.

about the hourly/daily stats, use cron.

Daenyth · 2010-02-23 15:09:05

Each of the tools I know does a slightly different thing, but try: nethogs, iftop, bwm-ng, ntop

johan84 · 2010-02-23 20:20:49

Sin.citadel wrote:

yes, it is right. but you can do any combination of rules to get the proper result, i myself have done this for an ip-specific rule so i can get the up/down usage for each of my network machines
also, you should use the -x parameter with iptables so all values are in bytes instead of MB/GB/KB, this makes it easier so you can simply add the values instead of converting them and then adding. Afterword you can use the -Z parameter with your specific chain to reset all the bytes for the rules.
i have a script for parsing with bash, and a python script which takes those values and puts them in a mysql database. if anyone needs it, i will upload it here.
about the hourly/daily stats, use cron.

the -Z I used allready but I didn't know about the -x (and that is indeed handy if you want to use the data afterwards)
If it's not too much trouble to post the script, I would really appreciate that.
Once in mysql there's no problem to make stats/graphs/totals (hour/month/...)

Is it maybe possible to get the current speed from the iptables?
To make a graph in conky I use

${color #0077ff}Down:${color lightgrey} ${downspeed eth0} k/s $alignr${color #0077ff} Up:${color lightgrey} ${upspeed eth0} k/s

That's why I first thought of creating a "virtual lan" where I would forward all local traffic to and another "internet lan" to forward all my internet traffic to (so I could also make graphs)
But I'll start with the rules I have know (as the statistics are more important to me now)

Thanks ;-)

Last edited by johan84 (2010-02-23 20:45:30)

johan84 · 2010-02-23 21:06:45

Daenyth wrote:

Each of the tools I know does a slightly different thing, but try: nethogs, iftop, bwm-ng, ntop

I came across ntop allready, but it has many functions and I couldn't find the totals lan/internet.
Maybe it is possible to create such statistics with the program, but I don't know how (yet).

Nethogs lets you know the stats per application, but that's not really what I'm looking for. (I would have to filter the apps and determine wich app is local/internet related)

iftop keeps total stats (realtime) don't really know if it can save automatically (I did saw a filter possibility, but have to figure out how to use it)

bwm-ng also keeps total stats (www and lan together).

I could be mistaken by some apps. I only tried them quickly. (On windows I use the bwmeter app, I was looking for a similar app)

iptraf can also save logfiles, but it saves the data seperately for each port (TCP/UDP)

I also tried mrtg (but I can't get the cfgmaker to work so I can't test the app: "Can't locate object method "initialize" via package "Pod::Usage" at /usr/bin/../lib/mrtg2/Pod/Usage.pm line 516.")

Thanks for the suggestions, my best bet would be on ntop to get it to work properly (what I want is actually much simpler than what ntop allready does)
Downloadd also keeps stats on IP-basis/protocol, but that's too detailed. I just need total internet/lan, and if possible current speed in graph/text (in terminal/conky doesn't matter)

iptables is allready a good help for my totals, I could use the data for speed too maybe, but my guess is that this would use too many cpu (I would have to check it at least almost every second to get a fluent graph)

FeatherMonkey · 2010-02-23 22:22:08

I've not seen vnstat mentioned but for the five minute average vnstat -tr 600 should do it. Not sure on the last but as it seems to log via interface it should be easy to distinguish between lan and internet.

Last edited by FeatherMonkey (2010-02-23 22:22:21)

johan84 · 2010-02-24 08:13:50

FeatherMonkey wrote:

I've not seen vnstat mentioned but for the five minute average vnstat -tr 600 should do it. Not sure on the last but as it seems to log via interface it should be easy to distinguish between lan and internet.

Internet and lan go through the same interface. So vnstat gives the sum of the two.
vnstat would indeed be a solution if I had internet and lan through different interfaces, but that's not the case.

That's why I thought maybe virtual interfaces could be the solution: creating a lan and an internet interface and using vnstat. (but I don't know if that's possible)

Sin.citadel · 2010-02-24 10:08:41

my setup contains 4 extra chains,

INPUT_LAN
OUTPUT_LAN
FORWARD_IN_LAN
FORWARD_OUT_LAN

the scripts below add the data for each ip from INPUT_LAN and FORWARD_IN_LAN for upload traffic and the other two chains for the download traffic and then store it into mysql.

as for current speed, u can write a script which takes the data after every second for 5 seconds, then subtract and then take an average to get rate of speed, but this is only useful if you need to take a snapshot after every hour or so, for realtime speed monitoring, use the tools pointed out by Daenyth.

#!/bin/bash

declare -r IPTABLES=/usr/sbin/iptables
declare -r LOG_IN="/tmp/input_traffic.log"
declare -r LOG_OUT="/tmp/output_traffic.log"
declare -r LOG_F_IN="/tmp/forward_in_traffic.log"
declare -r LOG_F_OUT="/tmp/forward_out_traffic.log"


$IPTABLES -nvx -L INPUT_LAN | tr -s [:blank:] | grep "192." |  cut -d' ' -f3,9,12 > $LOG_IN
$IPTABLES -nvx -L OUTPUT_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,10 > $LOG_OUT

$IPTABLES -nvx -L FORWARD_IN_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,9,12 > $LOG_F_IN
$IPTABLES -nvx -L FORWARD_OUT_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,10 > $LOG_F_OUT

$IPTABLES -Z

./dbTrafficLogging.py $LOG_IN $LOG_OUT $LOG_F_IN $LOG_F_OUT

rm $LOG_IN $LOG_OUT $LOG_F_IN $LOG_F_OUT

#!/usr/bin/python

# This Script Reads the Input/Output Bytes for each iptables rule for IP's
# And stores there relevant data information into their respective
# account Id's.

import MySQLdb as sql
import sys
if (len(sys.argv) < 4):
    # raise any error
    raise AssertionError,"4 Filenames Required"
conn = sql.connect('localhost','username','password',db='userdb')
cursor = conn.cursor()

inetData = {}
inputFile = open(sys.argv[1],'r')
outputFile = open(sys.argv[2],'r')
forwardInFile = open(sys.argv[3],'r')
forwardOutFile = open(sys.argv[4],'r')

for val in inputFile.readlines():
    val = val.split()
    inetData[val[1]] = [int(val[0]),0,val[2]]

for val in forwardInFile.readlines():
    val = val.split()
    inetData[val[1]][0] = inetData[val[1]][0] + int(val[0])

for val in outputFile.readlines():
    val = val.split()
    inetData[val[1]][1] = int(val[0])

for val in forwardOutFile.readlines():
    val = val.split()
    inetData[val[1]][1] = inetData[val[1]][1] + int(val[0])

for ip,data in inetData.iteritems():
    query = """UPDATE users SET 
    DataRecieved=DataRecieved+%d,DataSent=DataSent+%d WHERE IP="%s" """ % (data[1],data[0],ip)
    cursor.execute(query,)
conn.commit()
cursor.close()
conn.close()

inputFile.close()
outputFile.close()
forwardInFile.close()
forwardOutFile.close()

Gen2ly · 2010-02-25 02:08:03

I remember I tried this before and the only think I could find was netperf although these are some pretty good ideas.

http://bbs.archlinux.org/viewtopic.php? … 04#p657404

Lazze · 2010-02-25 12:51:48

Thanks for the huge amount of replies, sorry for my absence, I've been a bit busy lately and haven't had the time to try your suggestions out. I promise I'll give them a try as soon as possible.

Arch Linux

#1 2010-02-22 16:15:13

Script to get bandwidth usage

#2 2010-02-22 16:49:59

Re: Script to get bandwidth usage

#3 2010-02-22 19:15:47

Re: Script to get bandwidth usage

#4 2010-02-22 19:22:32

Re: Script to get bandwidth usage

#5 2010-02-22 21:59:41

Re: Script to get bandwidth usage

#6 2010-02-23 04:32:45

Re: Script to get bandwidth usage

#7 2010-02-23 15:09:05

Re: Script to get bandwidth usage

#8 2010-02-23 20:20:49

Re: Script to get bandwidth usage

#9 2010-02-23 21:06:45

Re: Script to get bandwidth usage

#10 2010-02-23 22:22:08

Re: Script to get bandwidth usage

#11 2010-02-24 08:13:50

Re: Script to get bandwidth usage

#12 2010-02-24 10:08:41

Re: Script to get bandwidth usage

#13 2010-02-25 02:08:03

Re: Script to get bandwidth usage

#14 2010-02-25 12:51:48

Re: Script to get bandwidth usage

Board footer