Using iptables to monitor bandwidth

MrSako · 09-13-2006, 03:23 PM

i need to generate basic reports of my bandwidth usage, how much data uploaded and how much downloaded.

i know this can be done on iptables and this article kind of briefly goes over it

http://www.linux.com/article.pl?sid=05/12/15/177232

but i don't really understand it, i dont think it directly applies to how i want to use it. I want to keep track of all my packets going in and out and seperate the usage by the interface (lo and venet0).

in addition to this I'd like to be able to rotate my logs on a certain day of every month (i am guessing this is something that needs to be a cronjob though and i can probally figure it out later)

could someone maybe give me so good hints/clues to what to do?

unSpawn · 09-13-2006, 04:18 PM

Moved: This thread is more suitable in the Networking forum and has been moved accordingly to help your thread/question get the exposure it deserves.

* Also look at the bottom of this page in the "Similar Threads" section. You *may* find some clues there.

theNbomr · 09-13-2006, 04:57 PM

Do you presently use iptables for firewalling or other purposes? Can you post the output from

Code:

'iptables -L'

I presently use iptables, along with a couple of cron jobs that collects stats from iptables, condenses it into a log file, displays it on a web page, and rotates the log weekly. My version is home-brewed, but quite simple, although not extremely polished. At present, it aggregates the total throughput, but could be adapted to discrete streams in and out.

I recently engaged another forum user on the subject here: http://www.linuxquestions.org/questi...d.php?t=478931

--- rod.

MrSako · 09-13-2006, 06:55 PM

i use iptables for basic port blocking.

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug pre fix `BANDWIDTH_IN:'
ACCEPT all -- anywhere anywhere state RELATED,ESTAB LISHED
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere multiport dports ft p,ssh,smtp,http,pop3,10000,3784,14534,51234 state NEW
ACCEPT udp -- anywhere anywhere multiport dports do main,3785,8767 state NEW
LOG all -- anywhere anywhere LOG level warning p refix `INPUT DROP: '

Chain FORWARD (policy DROP)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug pre fix `BANDWIDTH_OUT:'
LOG all -- anywhere anywhere LOG level debug pre fix `BANDWIDTH_IN:'

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug pre fix `BANDWIDTH_OUT:'

all that log stuff i beleive is from when i tried to activate the bandwidth monitor that comes with webmin, but it still has not ever generated any reports. i dont know if this is a fault in the webmin module or if something about my iptables setup is casuing it not to create logs that the bandwidth monitor is expecting to use (i dont know how the module works i do have a feeling it does the same thing im trying to do becasue it asks what type of firewall i have in the module config)

i think iptables is supposed to write to /var/log/bandwidth or /var/log/iptables (i forget) i have both these files exist but are empty. (i think i may have created the iptables file when attempting to do this a while back i forget)

edit: i forgot to write i think it's more likely becasue a default setting for the webmin bandwidth monitor is "Directory for bandwidth data Default (/etc/webmin/bandwidth/hours) " and hours does not exist in that location so perhaps its no iptables related.

im more interested in creating something not linked with webmin just for the expierence and also something i can more easily copy to another computer that isn't running webmin

theNbomr · 09-14-2006, 09:14 AM

Sorry, I was typing before I got thinking. I should have asked for 'iptables -vnxL'. A sample from the computer I am at right now looks like this:

Code:

Chain INPUT (policy ACCEPT 213486970 packets, 53252512207 bytes)
    pkts      bytes target     prot opt in     out     source               destination
      32     2036 DROP       all  --  *      *       147.52.136.195       0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 217876074 packets, 29218583380 bytes)
    pkts      bytes target     prot opt in     out     source               destination

As you can see, the number of bytes in and out are easy to distinguish. The basic approach I use to distill this information is to grab, with perl (but any number of other tools would work, too), anything that looks like /[0-9]+\s+bytes/, and sum them. For the iptables setup here, this bit of perl code seems to work:

Code:

#! /bin/perl -w
#
#    parseIptables.pl
#
use strict;
my $inBytes = 0;
my $outBytes = 0;
    while( <> ){
        if( $_ =~ m/Chain OUTPUT/ ){
            $_ =~ m/([0-9]+)\s+bytes/;
            $outBytes += $1;
        }
        elsif( $_ =~ m/Chain INPUT/ ){
            $_ =~ m/([0-9]+)\s+bytes/;
            $inBytes += $1;
        }
    }
    print "In: $inBytes, Out: $outBytes\n";

It would be used like this:

Code:

iptables -vnxL | parseIptables.pl

Add the -Z switch to the iptables commandline, and put the whole business into a cron job with the output redirected to a log file:

Code:

0-59 * * * * /sbin/iptables -vnxZ -L | parseIptables.pl >> /var/log/iptablesStats.log

The above will give a minute-by-minute snapshot of your throughput; both input and output. Of course, as I say, the parser code needs to be tailored to the specific iptables setup. It looks like this should work for your setup.

--- rod.

MrSako · 09-14-2006, 03:57 PM

thanks i think ill try that out

that code goes ni a file called parseIptables.pl right? (with executable privledges)

i don't really understand this last line

0-59 * * * * /sbin/iptables -vnxZ -L | parseIptables.pl >> /var/log/iptablesStats.log

whats the 0-59 * * * * ?

theNbomr · 09-14-2006, 05:18 PM

Yes, it is a perl script. Either put it somewhere that is in the search path for cron, or specifiy the full filespec to it. THe last part is a crontab entry. This should be entered into the root crontab (as root, 'crontab -e'). The '0-59 * * * *' part is the specifier for the times/dates that the cron job will run.

man 5 crontab

--- rod.

MrSako · 09-14-2006, 06:24 PM

i just made that parseIptables.pl file with your code when i tried to run it i got this error. (it had exectuable privledges)

# iptables -vnxL | parseIptables.pl
-bash: parseIptables.pl: command not found

unSpawn · 09-14-2006, 06:36 PM

-bash: parseIptables.pl: command not found
Means it's not in your $PATH. If you saved it in /tmp and made it executable you do:
"iptables -vnxL | /tmp/parseIptables.pl", like that or save it in a directory that's in your PATH statement (echo "$PATH").
The next error most likely will be about the path inside the script since your Perl will reside say in /usr/bin, so change the shebang line (first line) to read "#!/usr/bin/perl -w", no spaces except between "l" and "-".
The next error most likely will be about the add part.

MrSako · 09-14-2006, 06:54 PM

thanks...

ok here is what my file looks like now which wont compile right
.. first the error

Quote:

# iptables -vnxL | ./parseIptables.pl
syntax error at ./parseIptables.pl line 15, near "else if"
syntax error at ./parseIptables.pl line 20, near "}"
Execution of ./parseIptables.pl aborted due to compilation errors.

Code:

#! /usr/bin/perl -w
#
#       parseIptables.pl
#
use strict;
my $inBytes = 0;
my $outBytes = 0;
  while( <> )
  {
    if( $_ =~ m/Chain OUTPUT/ )
    {
        $_ =~ m/([0-9]+)\s+bytes/;
        $outBytes += $1;
    }
    else if( $_ =~ m/Chain INPUT/ )
    {
        $_ =~ m/([0-9]+)\s+bytes/;
        $inBytes += $1;
    }
  }
  print "In: $inBytes, Out: $outBytes\n";

i know else if used to be one word but i got this erorr before making it two words

Quote:

# iptables -vnxL | ./parseIptables.pl
elseif should be elsif at ./parseIptables.pl line 15.
syntax error at ./parseIptables.pl line 16, near ")
{"
syntax error at ./parseIptables.pl line 20, near "}"
Execution of ./parseIptables.pl aborted due to compilation errors.

i presume writing it as one word was a typo

theNbomr · 09-14-2006, 11:53 PM

No, one word. Why don't you cut and paste from the code box? That's the whole point; formatting is retained. The code as posted was tested and ran correctly. I just corrected the one 'else-if' mistake, and it runs perfectly.

--- rod.

jaz_comp · 09-15-2006, 01:01 AM

Installing Iptables Log Analyzer:

Iptables Log Analyzer is a package that analyzes the log output from your iptables firewall, stores the info in a database and then produces a nice user friendly web interface from where you can monitor your firewall log output at any time. The interface even lets you sort throught the logs and group logs by category. The official site for Iptables Log Analyzer can be found at http://www.gege.org/iptables/.

So let's install it...

The first thing you will need to do is to download the Iptables Log Analyzer package. You can download the latest copy of the package right here: http://www.iptablesrocks.org/downloa...er_v0.4.tar.gz

Now unpack it...

tar zxvf iptables_logger_v0.4.tar.gz

And now let's start the installation...

cd iptables

Log into mysql and do the database work...

mysql -u root -p

create a database called "iptables"...

mysql> create database iptables;

Now create an admin user for the database...

mysql> grant all privileges on iptables.* to iptables_admin@localhost identified by 'xxxxx';

And a database user for the php interface...

mysql> grant all privileges on iptables.* to iptables_user@localhost identified by 'xxxxx';

All done. Let's exit out.

mysql> quit

Now import the "iptables" database information into the database...

cat sql/db.sql | mysql -u iptables_admin -p iptables

OK, the database should be ready now. Next we'll Install the web interface.

cp -R web /path/to/webdocs/directory/firewall (Example: cp -R web /var/www/html/firewall)

Now you will need to configure the web interface. This is done via the "config.php" file within the interface files.

vi /path/to/webdocs/directory/firewall/config.php (Example: vi /var/www/html/firewall/config.php)

Make sure your configuration data is as follows:

# Host of the MySQL database
$db_host="localhost";

# User of the MySQL database
$db_user="iptables_user";

# Password of the MySQL database
#Make sure you enter your "iptables_user" password in place of the red x's below
$db_password="xxxxxx";

# Name of the database
$db_name="iptables";

# URL Path to your installation
$url_base="/firewall/";

#debug mode
$debug=1;

#The default number of record displayed
$default_number=20;

#The default chain displayed
$default_chain="ALL";

#The default date for packets (10000 means any)
$default_date=10000;

#The default ignored ports
$default_ignored_ports= array();

Save and exit the file.

Now we will configure and install the database feeder script.

vi scripts/feed_db.pl

Make sure the following configuration section is set properly. Make sure you enter the "iptables_user" mysql password where the x's are.

my $dsn = 'DBI:mysql:iptables:localhost';
my $db_user_name = 'iptables_admin';
my $db_password = 'xxxxx';
my $log_file = '/var/log/firewall';
my $pid_file = "/var/run/iptablelog.pid";

Save and exit the file.

Now we will copy the database feeder script to a location from which it can be easily called:

cp feed_db.pl /usr/local/bin/

Next, copy the init.d script called "iptablelog" to your server's init.d directory..

cp scripts/iptablelog /etc/rc.d/init.d/

Note: I've already pre-configured the "iptableslog" init.d script, but you may want to customize it further depending on your needs.

So, let's start it up!
/etc/init.d/iptablelog start

Alright! That should be it. Iptables Log Analyzer should now be running! So, let's take a look at the web interface.

Open a browser and go to: http://www.yourdomain.com/firewall

You should get the Iptables Log Analyzer screen. It's pretty self explanatory, so just play around with it for a while and you'll figure it out. If you don't see any logs yet, you can trigger a test by logging out of your server and then logging in again via SSH. The SSH connection will be recorded in the iptables logs and this will appear on the Iptables Log Analyzer screen. If you need more help with Iptables Log Analyzer, check out the homepage at http://www.gege.org/iptables/.

In the next step, we'll cover firewall maintenence as well as step to modify your firewall when needed. We'll also go over how to ensure that your firewall starts up on boot.
Thanks & regards

Jaz_comp
RHCE

theNbomr · 09-15-2006, 09:07 AM

Looks slick. A comment & a couple of questions...

Iptables log analyzer is much more than a bandwidth monitor, which is what the original poster asked for.

Does the system rely upon a specific configuration of the iptables chains and tables? What if I already have a configuration that I use for specific purposes?

How long are the logged records retained in the SQL database? My firewalls log a lot of traffic, and if the logs are retained indefinitely, would result in huge databases. Is the importation of logged records done in real-time, or on a periodic batch basis?

I haven't inspected the perl code that extracts logs into the database, but do you know how much MySQL-specific code is in there? I would want to use postgresql.

--- rod.

MrSako · 09-15-2006, 04:26 PM

Quote:

Originally Posted by theNbomr

No, one word. Why don't you cut and paste from the code box? That's the whole point; formatting is retained. The code as posted was tested and ran correctly. I just corrected the one 'else-if' mistake, and it runs perfectly.

--- rod.

i can't copy and paste.

the only accsss i have to the server is shell. its a VPS im renting im not actually sitting at this computer. i guess i could copy and paste it on my computer here and upload it ill try that

btw could someone tell me the command for being on ssh and you want to send a file to the computer the ssh client is connected to from the computer your on? i currently use a seperate ssh program to do this becasue it has a user interface like a FTP client. itd be cool to just type it in and not have to open a whole new thing (i know this is off topic my mistake)

MrSako · 09-15-2006, 04:37 PM

alright, so in perl its elsif (not else its els)

fixed everything and it works sweet :-d

though i guess it's not working how i thought it would. I want to do stuff to generate a log like

--start septemeber 12th
monitor ran @ whatevertimee - In: 1234 Out: 1234
monitor ran @ whatevertime2 - In: 1235 Out: 1236
--start october 12th
monitor ran @ whatevertime3 - In: 5 Out: 10
monitorran @ whatevertime4 - In:10 Out:20

etc

my allowed bandwidth is reset every month usually on the 12th but not always.

so what needs to happen is the script has to search the iptables output for packets sent inbatween certain dates etc to do what i want right? (not sure how to do this >.<)

and also a seperate script i can run at the begining of every "month" where the bandwidth gets reset.

If someone can tell me how to search the iptables output for packets inbatween date x and date y thatd be great, id love to figure the rest out on my own.

thanks for all your helps