LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 04-08-2007, 05:58 PM   #1
ErV
Senior Member
 
Registered: Mar 2007
Location: Russia
Distribution: Slackware 12.2
Posts: 1,202
Blog Entries: 3

Rep: Reputation: 62
Traffic accounting system for Slackware


Hello!
I've been looking for a traffic accounting system (for Slackware 11) for a certain amount of time. I originally needed a hour/day/week/year statistics sorted by a certain ipranges (for a different iptable chains). Now I made my own system on bash scripts+bc+crond+iptables. I've decided to post it here - I just hope someone will need it and will find it useful. I'd like to hear some comments/complaints/hints for optimizations/questions. So, here it is:

System consists of three scripts. rc.firewall, /usr/lib/vsi-traf/traffic.sh /usr/lib/vsi-traf/iptable.pl (modified version, copyrights are intact). I also wrote some documentation, it's also here. (To use script, You'll need bc)

Here they are:
rc.firewall
Code:
#!/bin/sh
#firewall script
#sample firewall script originally used with 
#traffic.sh

echo firewall script started
CARD_MASK="192.168.1.2/32"
ROUTER="192.168.1.1/32"
BVF="84.17.243.19/32"
RANGE1="77.45.128.0/255.255.128.0"
RANGE2="80.82.32.0/255.255.224.0"
RANGE3="88.83.192.0/255.255.192.0"

FREE[0]=$RANGE1
FREE[1]=$RANGE2
FREE[2]=$RANGE3
FREE[3]=""

KILL[0]="80.82.32.10/32"
KILL[1]="80.82.32.19/32"
KILL[2]=""

PAID[0]="80.82.32.27/32"
PAID[1]="80.82.32.11/32"
PAID[2]=""

LOCAL="127.0.0.1/32"

iptables --flush
iptables -X

iptables -N local_in
iptables -N local_out
iptables -N extern_in
iptables -N extern_out
iptables -N total_in
iptables -N total_out
iptables -N kill_in
iptables -N kill_out

iptables -N kill_all
iptables -N extern_all
iptables -N local_all
iptables -N total_all

iptables -A local_all -j ACCEPT
iptables -A extern_all -j ACCEPT
iptables -A kill_all -j DROP
iptables -A total_all -j RETURN

iptables -A total_in -j total_all
iptables -A total_out -j total_all
iptables -A local_in -j local_all
iptables -A local_out -j local_all
iptables -A extern_in -j extern_all
iptables -A extern_out -j extern_all
iptables -A kill_in -j kill_all
iptables -A kill_out -j kill_all

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

iptables -A INPUT -s $LOCAL -d $LOCAL -i lo -j ACCEPT
iptables -A OUTPUT -s $LOCAL -d $LOCAL -o lo -j ACCEPT

iptables -A INPUT -s $ROUTER -d $CARD_MASK -j ACCEPT
iptables -A OUTPUT -s $CARD_MASK -d $ROUTER -j ACCEPT

i=0
while [ "${KILL[$i]}" != "" ]
do
    iptables -A INPUT -p all -s ${KILL[$i]} -j kill_in
    iptables -A OUTPUT -p all -d ${KILL[$i]} -j kill_out
    i=$(($i+1))
done

iptables -A INPUT -p all -j total_in
iptables -A OUTPUT -p all -j total_out

iptables -A INPUT -p tcp -s $BVF -d $CARD_MASK -j local_in
iptables -A OUTPUT  -p all -s $CARD_MASK -d $BVF -j local_out

i=0
while [ "${PAID[$i]}" != "" ]
do
    iptables -A INPUT -p all -s ${PAID[$i]} -j extern_in
    iptables -A OUTPUT -p all -d ${PAID[$i]} -j extern_out
    i=$(($i+1))
done

i=0
while [ "${FREE[$i]}" != "" ] 
do
    iptables -A INPUT -p all -s ${FREE[$i]} -d $CARD_MASK -j local_in
    iptables -A OUTPUT -p all -s $CARD_MASK -d ${FREE[$i]} -j local_out
    i=$(($i+1))
done

iptables -A INPUT -p all -j kill_in
iptables -A OUTPUT -p all -j kill_out

#iptables -A INPUT -p icmp -j ACCEPT
#iptables -A POSTROUTING -p all -s 127.0.0.1 -d 80.82.32.27 

CARD_MASK=""
RANGE1=""
RANGE2=""
RANGE3=""
LOCAL=""
traffic.sh
Code:
#!/bin/sh
# This is script is provided "as is", without warranty of any kind.
# You use it at your own risk. In no event author may be liable for any 
# damage done by it's use/misuse. :)
# It can be modified and distributed freely, as long as the 
# author remain mentioned here.
#
# Made by ErV (email address removed)

BASE=/usr/lib/vsi-traf
DATA=$BASE/data
HOUR=`date +%H`
DAY=`date +%d`
WEEK=`date +%W`
WEEK=$(($WEEK%4))
MONTH=`date +%m`
YEAR=`date +%Y`
LOCK_FILE="$BASE/lock"
MONTH_DIR="$YEAR/$MONTH"
WEEK_DIR="$MONTH_DIR/weeks"
DAY_DIR="$MONTH_DIR/$DAY"
BYTES_HOUR="$DAY_DIR/$HOUR:00-$HOUR:59"
BYTES_DAY="$MONTH_DIR/$DAY.total"
BYTES_WEEK="$WEEK_DIR/$WEEK.total"
BYTES_MONTH="$YEAR/$MONTH.total"
BYTES_YEAR="$YEAR.total"
CURRENT_HOUR="this_hour"
CURRENT_DAY="this_day"
CURRENT_WEEK="this_week"
CURRENT_MONTH="this_month"
CURRENT_YEAR="this_year"
GET_BYTES=$BASE/iptable.pl

CHAINS[0]="extern_in"
CHAINS[1]="extern_out"
CHAINS[2]="local_in"
CHAINS[3]="local_out"
CHAINS[4]="total_in"
CHAINS[5]="total_out"
CHAINS[6]=""

force_directory(){
    #echo "$1"
    if [ ! -r "$1" ]; then
	mkdir "$1";
    fi;
}

#arg 1 - fuction that accepts chain name
for_each_chain(){
    i=0;
    while [ "${CHAINS[$i]}" != "" ]; do
	$1 ${CHAINS[$i]};
	i=$(($i+1))
    done;
}

check_chain_dir(){
    force_directory "$DATA/$1"
    force_directory "$DATA/$1/$YEAR"
    force_directory "$DATA/$1/$MONTH_DIR"
    force_directory "$DATA/$1/$WEEK_DIR"
    force_directory "$DATA/$1/$DAY_DIR"
}

force_counter(){
    if [ ! -r "$1" ]; then
	echo "0" > "$1";
    fi
#    if [ `cat "$1"` == "" ]; then
#	echo "0" > "$1";
#    fi
}

check_chain_counters(){
    CHAIN_DIR="$DATA/$1"
    force_counter "$CHAIN_DIR/$BYTES_HOUR"
    force_counter "$CHAIN_DIR/$BYTES_DAY"
    force_counter "$CHAIN_DIR/$BYTES_WEEK"
    force_counter "$CHAIN_DIR/$BYTES_MONTH"
    force_counter "$CHAIN_DIR/$BYTES_YEAR"
    ln -s -f "$CHAIN_DIR/$BYTES_HOUR" "$CHAIN_DIR/$CURRENT_HOUR"
    ln -s -f "$CHAIN_DIR/$BYTES_DAY" "$CHAIN_DIR/$CURRENT_DAY"
    ln -s -f "$CHAIN_DIR/$BYTES_WEEK" "$CHAIN_DIR/$CURRENT_WEEK"
    ln -s -f "$CHAIN_DIR/$BYTES_MONTH" "$CHAIN_DIR/$CURRENT_MONTH"
    ln -s -f "$CHAIN_DIR/$BYTES_YEAR" "$CHAIN_DIR/$CURRENT_YEAR"
}

check(){
    force_directory $DATA
    for_each_chain check_chain_dir
    for_each_chain check_chain_counters
}

#$1 is a counter file $2 is a value to add
inc_counter(){
    PREV_BYTES=`cat $1`
    echo "$PREV_BYTES + $2"|bc > $1;
}

#$1 is a chain name, $2 is a new added value
chain_inc_counters(){
    BYTES="$2"
    CHAIN_DIR="$DATA/$1"
    inc_counter "$CHAIN_DIR/$BYTES_HOUR" "$BYTES"
    inc_counter "$CHAIN_DIR/$BYTES_DAY" "$BYTES"
    inc_counter "$CHAIN_DIR/$BYTES_WEEK" "$BYTES"
    inc_counter "$CHAIN_DIR/$BYTES_MONTH" "$BYTES"
    inc_counter "$CHAIN_DIR/$BYTES_YEAR" "$BYTES"
}

chain_update_normal(){
    BYTES=`$GET_BYTES $1`
    chain_inc_counters "$1" "$BYTES"
}

lock(){
    echo "0" > "$LOCK_FILE"
}

unlock(){
    rm "$LOCK_FILE"
}

update_normal(){
    if [ ! -r "$LOCK_FILE" ]; then
	lock;
        check;
	for_each_chain chain_update_normal;
	unlock;
    else
	echo another instance already working
    fi;
}

case $1 in
    'check') check;;
    'update') update_normal;;
    'unlock') unlock;;#use only after system restart!!!
    'lock') lock;;
    *) echo unknown argument
	echo "usage traf [update|check|lock|unlock]"
esac
iptable.pl
Code:
#!/usr/bin/perl 
## 
## This is a quick perl script to 
## pull bandwidth usage from iptables chains 
## 
## If you use/optimize this script, please let me know. 
## Brian Stanback : brian [at] stanback [dot] net 
# 
## Example iptables rule for web bandwidth usage: 
## > iptables -N WWW 
## > iptables -A WWW -j ACCEPT 
## > iptables -A INPUT -p tcp -m tcp --dport 80 -j WWW 
## > iptables -A OUTPUT -p tcp -m tcp --sport 80 -j WWW 
## 
## Run "iptables.pl WWW" as root to test, note that you can 
## combine more than one protocol into a single chain. 
## 
## Sudo Configuration (/etc/sudoers) 
## > www-data    ALL = NOPASSWD: /usr/share/cacti/scripts/iptables.pl 
## 
## The Input String should be set to "sudo <path_cacti>/scripts/iptables.pl <chain>" 
## and you will need to setup an input field so that the <chain> argument can be passed. 
## 
## The data input type should be set to COUNTER 
## 
# 
# 
# modified by: Paul Campbell <forums@campbell-multimedia.co.uk> 
# Now returns a seperate entry for each rule.  Output for a 
# 3 rule chain might now be: 
# rule1:123 rule2:456 rule3:789 
#
# Modified by ErV. Now outputs number of bytes only.

if ($ARGV[0]) { 
        $chains = `/usr/sbin/iptables --line-number -xnvL $ARGV[0]`; 
	`/usr/sbin/iptables --zero $ARGV[0]`;
        @chains = split(/\n/, $chains); 
        shift(@chains); 
        shift(@chains); 
        foreach( @chains ) { 
                /(\d+)\W+[0-9]+\W+([0-9]+)\W+/; 
#                print " rule$1:$2"; 
		print"$2";
        } 
        print "\n"; 
        #$chains[2] =~ /[\W+]?[0-9]+\W+([0-9]+)\W+/; 
} else { 
        print "Usage: $0 Chain\n"; 
}
documentation:
Quote:
INTRO:
This is a free software , and it's provided without warranty of any kind.
You use it at your own risk. IN no event author may be responsible
for any damage done to your/your PC/ something else by proper/improper
use/misuse of this script. You can distribute it freely and use for any
purpose, as long as origin is not misrepresented and source is open.
(although it's difficult to close source for a script )

DOCUMENTATION:
This is a relatively simple script that helps you to gather statistics
from iptables, and organize it (by day/week/year/month) To use it effectively,
you must be familiar iptables and you must know what you need.

Normal usage is to invoke script at system startup with "unlock" argument,
to run it at least once in hour (using crond) with "update" argument and
to run it once again at system shutdown, with "update" argument.

Script will need file iptable.pl to work. This file must be placed in same
directory, and it must be executable. YOu'll also need bc and perl (and bash )
to make this script work.

Script must have write permissions to the directory it's placed into.
By default, it must be placed into /usr/lib/vsi-traf/ directory.
If you will (and you probably will ) chose another directory, you must
change BASE variable value to a correct path. (without slash at the end).

The iptable chain names must be placed in the CHAINS variable.
You must have an "empty string" variable after last entered name.
The number of names is not limited by anything.
For example, if you want to monitor "INPUT" and "OUTPUT" chains, you
should replace CHAINS declaration with:
CHAINS[0]="INPUT"
CHAINS[1]="OUTPUT"
CHAINS[2]=""
if you want to monitor FORWARD chain only, you'll need to change declaration
to this:
CHAINS[0]="FORWARD"
CHAINS[1]=""
and so on.

After you are done with declaring chains, you are ready to use script .
When started with "check" or "update" arguments it will create "data" folder
in its directory, and then it will create a subfolder for every chain you are watching.
Let's assume your chain is called simply "chain" So, following directories will be created.

(program_dir)/data/chain/ - root directory with statistics for this chain
(program_dir)/data/chain/2007 - a statistic for this chain during 2007 year.
(program_dir)/data/chain/2007.total - a total number of bytes that went through
this chain during year 2007. (ASCII representation of a number)
(program_dir)/data/chain/2007/04.total - a total number of bytes of april of 2007.
(program_dir)/data/chain/2007/04/ - a folder with statics for april.
(program_dir)/data/chain/2007/04/weeks - per-week statistic for april 2007/
(program_dir)/data/chain/2007/04/08.total - total number of bytes for 08 april of 2007.
(program_dir)/data/chain/2007/04/08/ - statistics for 08 april of 2007.
(program_dir)/data/chain/2007/04/08/23:00-23:59 - a number of bytes passed through this chain
during the period of 23:00-23:59 on 08 april of 2007.

(program_dir)/data/chain/this_year - a symbolic link to a current year's statistics
(program_dir)/data/chain/this_month - a symbolic link to a current month's statistics
(program_dir)/data/chain/this_week - a symbolic link to a current week's statistics
(program_dir)/data/chain/this_day - a symbolic link to a current day's statistics
(program_dir)/data/chain/this_hour - a symbolic link to a current hour's statistics.

Links are updated every time when script is called with either "update" or "check" parameters.

I think you got the idea. Although this is an unusual implementation (it eats disk space )
(I think) it's good for a Home PC that can be turned off at any time, so it can't gather
statistics (as the large servers do) once during the day. Since there is a separate counter
for every possible date, it difficult to use wrong counter, etc.Also, the script is really simple
and easy to use. It's also bullet-proff against many unusual situations due to it's algorithm.
It also takes (tries to take) multithreading in account - so it can't be called twice (although I
didn't test it) at same time, which means it can't garble statistics.

arguments:
1) unlock - removes lockfile that prevents from running additional copy.
2) lock - creates lockfile
3) check - creates all needed directories and checks counters. (check is called when an update
called, so it isn't really necesarry)
4) update - updates the counters.

Made by ErV.
You can contact me (if you really wish).
via erv000 <at> sourceforge.net.
here are also some Slackware-specific fragments:
rc.local_shutdown:
Quote:
#!/bin/sh
echo shutdown script started
/usr/lib/vsi-traf/traffic.sh update
rc.local:
Quote:
#!/bin/sh
#
# /etc/rc.d/rc.local: Local system initialization script.
#
# Put any local startup commands in here. Also, if you have
# anything that needs to be run at shutdown time you can
# make an /etc/rc.d/rc.local_shutdown script and put those
# commands in there.

/usr/lib/vsi-traf/traffic.sh unlock
/var/spool/cron/crontabs/root
Code:
# If you don't want the output of a cron job mailed to you, you have to direct
# any output to /dev/null.  We'll do this here since these jobs should run
# properly on a newly installed system, but if they don't the average newbie
# might get quite perplexed about getting strange mail every 5 minutes. :^)
#
# Run the hourly, daily, weekly, and monthly cron jobs.
# Jobs that need different timing may be entered into the crontab as before,
# but most really don't need greater granularity than this.  If the exact
# times of the hourly, daily, weekly, and monthly cron jobs do not suit your
# needs, feel free to adjust them.
#
# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 4:40 every day:
40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 4:30 on the first day of the week:
30 4 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#
# Run monthly cron jobs at 4:20 on the first day of the month:
20 4 1 * * /usr/bin/run-parts /etc/cron.monthly 1> /dev/null

*/5,59 * * * * /usr/bin/bash /usr/lib/vsi-traf/traffic.sh update > /dev/null
The script is fresh (I've completed it two hours ago), and can contain bugs. Anyway I'd like to know, If it is useful for someone, or if it can be improved...

EDIT:
real email addresses removed (due to spam).
All files are available at http://shtraf.sourceforge.net or http://sourceforge.net/projects/shtraf along with contact information and newer files. Project will be most likely abandoned in the future.

Last edited by ErV; 05-29-2008 at 11:24 AM.
 
Old 04-09-2007, 11:38 AM   #2
alibar
Member
 
Registered: Mar 2007
Posts: 44

Rep: Reputation: 15
Very clever!

What a handy little piece of script. I'm going to run it and have a closer look when i get back to base. If you wouldn't mind i might try a simple front end to tie in with it? Brilliant idea though, your a true pioneer!

Lead the way captain erV!

EDIT: how long did you spend on it?
 
Old 04-09-2007, 01:34 PM   #3
ErV
Senior Member
 
Registered: Mar 2007
Location: Russia
Distribution: Slackware 12.2
Posts: 1,202

Original Poster
Blog Entries: 3

Rep: Reputation: 62
Quote:
Originally Posted by alibar
If you wouldn't mind i might try a simple front end to tie in with it?
Of course I won't mind . It will be good to modify this script and made some installer for it or some report generator or anything... Anyway, its a OpenSource world, everyone is free to modify it for their needs I'll be happy if someone else will use it.

Quote:
Originally Posted by alibar
Brilliant idea though, your a true pioneer!

Lead the way captain erV!
Thanks. Glad to hear it. I'm glad you liked the idea, too

Quote:
Originally Posted by alibar
EDIT: how long did you spend on it?
Well.. (sorry if answer is too detailed ) I've been looking for something to replace Windows BW Meter detailed statistics since I switched to a linux (about a two months ago? I'm not sure). In the March I had enough knowledge to work with slackware comfortably, tuned it a bit and added simple firewall (firewall.rc). But there were no BW meter replacement. at 22th of March I asked on several forums if such replacement exists and didn't find anything that suits my needs. At 31th I found out a counters in iptables and started looking for a way extract their information (tried mrtg, for example, good utility, but without daily/hourly/monthly statistics). I wanted to write my own C++ utility - to gather a statistics, and to draw charts in realtime. But I was a bit unsure how it will behave in power failures, etc. Yesterday I was toying with bash scripts and suddenly understood that they can be used in the way similar to C/C++ and it'll be easy to extract data from iptables with perl script I've already found. I've been thinking for some time how to make an "easy" logfile that will not be vulnerable to switching machine on and off. I just wanted logfile that can be easily processed to compute totals, and that will be making only one record per hour, but will keep this record up-to-date when switching machine on and off. Then I've suddenly got an idea that instead of making logfile it will be much simpler to make a separate counter file for each counter, and they can be easily organized in herarchical DB-like structure, and that I can use bc to make counter precision infinite. Since the date changes when the script update is called, it will be invulnerable to switching machine on and off. So, here it is... Not sure how much time it took to make it. I wrote it in 1 or 3 hours (debugging, thinking, etc.). But I've been learning how this can be done during a several weeks...

Oh, and I almost forgot (it's probably obvious, but just in case). rc.firewall here is tuned for my personal needs, it blocks almost everything, and is just for reference/example how to do it (it splits traffic in several streams and make it flow through several different counters). To make another system/tune it for personal needs, someone will have to modify it rc.firewall. If someone will need explanation how it works, I'll write it.

P.S. Maybe(can't promise) I'll make some C++ program with same functionality later (if I will have time for this). I would be good (but certainly not neccesarry) to put data in a database instead of files - since small files takes some amount of hard-drive space (which is bigger than file), or add some report-generator, etc... But I think that script is better, since it is easily readble, it is easy to change, etc.

Last edited by ErV; 04-09-2007 at 01:47 PM.
 
  


Reply

Tags
accounting, network, slackware, traffic


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Ibm Japan Builds New Accounting System for NYK Line LXer Syndicated Linux News 0 06-17-2006 09:54 AM
IP Traffic accounting varun_saa Mandriva 0 02-26-2005 05:13 AM
iptables traffic accounting for each IP/MAC sheyh Linux - Networking 0 07-11-2004 04:55 PM
traffic accounting? how? sciobanu Linux - Networking 4 07-08-2004 11:09 AM
Traffic accounting...? Gorbachov Linux - Software 2 09-23-2003 07:16 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 12:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration