Monitor and Log Internet Bandwidth Usage
 

I'm looking for a simple way to monitor and log my internet bandwidth usage. Not total network device usage, just internet usage.

Something that provides a simple chart of daily, monthly, and yearly usage, but ignoring all bandwidth on my internal LAN.

I notice several possible tools, such as vnstat, ntop, iftop. Yet all of them seem focused on tracking the entire network interface. I want to ignore LAN usage. I do not really care about LAN bandwidth.

iftop seems intended only for on-the-fly usage and not cumulative logging. I can't tell whether vnstat or ntop can be configured to log only internet usage rather than all traffic through the network device.

I do not want to log every connection like squid. The utility should only log stats on a daily basis, but also be able to display cumulative totals from those daily entries.

I don't need DNS resolution, port monitoring, etc.

I prefer something that runs in the background as a service or daemon, but can provide statistics quickly with a terminal window. All I want is to view total daily, monthly, and yearly internet usage. Perhaps even pipe the output to a local email each day too.

Any help is again appreciated. :)

Many home routers provide this function. For example:

You can retrieve and log this information. You can reset the counters on a regular interval (hourly, daily, weekly, etc.) so that you can easily chart interval usage.

Please correct me if I'm wrong, but with your example all I would know is the number of packets, not the actual bandwidth. Additionally, many routers do not maintain history across a power cycle or don't organize the stats as I specified. If the router does not save the data, then I'd have to grab and store that information from the router in real-time if the router is not powered on 24/7.

All I want is a utility that performs a basic function similar to tcpdump:

tcpdump -i eth0 -t -n ip and not net localnet

But stored and organized in a manner that I can view the data in some kind of readable format:

The problem is that monitoring the size of each packet requires packet inspection which is resource intensive; most small router chipsets don't include the functionality.

You can perform the function with a script and tcpdump's '-G' option, processing the collected data after each rotation. As I said, it's resource intensive, which is why most applications don't perform this type of function.

You can check to see if your ISP provides bandwidth usage statistics (those that impose any type of cap usually do). Higher-end routers have the ASICs that can handle this functionality in hardware. If your ISP imposes no limitations or usage charge, then why dedicate resources to the function?

Is tcpdump a viable method to provide myself the information I seek?

I have a dual core 2.3 GHz system with SATA II drives. How much impact would I notice?

Back to my original post: are there any command line apps that already do all of this?

Short answer: Curiosity. :)

Longer answer: Small mom-and-pop WISP located in the rural boonies. More users are discovering video streaming and VOIP. Bandwidth usage is outpacing customer growth. As customers consume more bandwidth the amount the owner pays to the backhaul provider rises. Bottom line is I think caps are coming. Would be nice to know ahead of time what I use.

I have been monitoring with knemo. I like the concise presentation of the data and the data is stored in a simple XML-like text file. Yet knemo cannot distinguish between the internet and LAN. The other thing I don't like about knemo is I have to be running X/KDE.

I have a Linksys WRT54GL 1.1 router with DD-WRT v23SP2 firmware. Has worked well a long time but there are no usage stats.

I was considering updating to a newer version. I am not so sure DD-WRT is the right decision anymore (the developers have written instructions so full of warnings and anal details that I feel like I am studying how to fly a rocket to the moon).

I was thinking about trying Tomato. One way or another I won't update unless I know the usage data is saved during a power-off. I do not keep my systems or router powered 24/7. The knemo app stores data efficiently so I don't know why router firmware wouldn't.

Monitoring with the router does make sense, but until I draw enough courage to flash the router I figured tracking usage from my computer is just as good. Hence my post. :)

The impact is a function of the number of packets that have to be examined. 50/sec; no problem. 100,000/sec; can't do it in software on a fast desktop machine anymore. That's why the corporate-type routers have ASICs (application specific integrated circuits) to perform the function.

If you just want to get an idea, fire up wireshark, start capture on your interface (e.g. eth0), select menu Statistics->IO Graph. Unclick the 'Graph 1' button, and enter a filter like:

where "123.45.67.0/24" is your LAN subnet. Then select the 'Graph 1' button to start plotting the traffic. Watch your CPU utilization to get an idea of the resource involved for your load.

Possibilites!
 

I found a few tools some years back that may be helpful to you.

stager - software.uninett.no/stager

flow-tools - www.splintered.net/sw/flow-tools/

flowscan - net.doit.wisc.edu/~plonka/FlowScan/

I have used flow-tools and flowscan to collect and graph data thru rrd. I haven't used stager, but it looks like a really good tool. I was planning on implementing stager, before I left a former employer.

Of course they may all be beyond your needs...

Since I don't know your network design, it is hard to make proper recommendations. I have used ntop in the past, and it provided good information to me. I believe that you can define criteria such as home network and external network. You may just have to play with a few different tools to see, which will work best for you.

Enjoy!

KNemo works very well for me. It also saves the stats in a file in your /home directory.

http://www.kde-apps.org/content/show.php?content=12956

These are basic home machines. My office machine does regular file transfers with rsync, but nothing dramatic.

I've been using KNemo, but as I shared in a previous post, the app provides no way to distinguish between internet traffic and LAN traffic. :( I do like the simple chart.

I have 4 machines connected to a D-Link DGS-2205 Gigabit switch, which is connected to Linksys WRT54GL 1.1 router, which connects to the WISP Subscriber Unit. Two machines are powered on only seldom, two are powered on often. Srictly speaking, I should monitor internet traffic at the router because the latter two are on so often. I've been trying to learn more about Tomato and DD-WRT with respect to saving bandwidth history at the router, but I have not found anything conclusive. I don't know why that would be so hard to implement when KNemo uses a simple text file that is only a few KB in size.

I found lots of information about using tools like Cacti and MRTG extracting data from the router through SNMP, all of which is over my head and makes my eyes water. :)

However, I have been wondering if the solution might be simple. As I only want to monitor total throughput to the internet, the router can provide that data either through ifconfig vlan1 or /proc/net/dev. The trick is to save that data somewhere.

Except in unique circumstances, I always power down my machines at night as well as my switch/router. I do not have anything related powered on 24/7, which adds to the challenge. Another challenge is the router has no shutdown sequence, just on and off.

The DD-WRT firmware supports scp. Maybe a cron job script run every minute to send the data in a text file to any machine on the LAN found powered on. From there I could manipulate the data.

Any ideas?

Maybe you could try vnstat:

There is a Slackbuild for this in the network section, easily searched by sbopkg. The Readme has the info.

vnStat, as with most of these suggestions, doesn't have the ability to only report on a subset of the traffic. They don't meet the OP requirements.

I'm not a network expert by any stretch of the imagination. I was searching for something to help and came across this. If it can selectively monitor any interface, then exactly why won't it work for the OP? Why can it not be configured to only monitor router traffic to and from the Internet?

I wish I had two machines to test this.

I also see in the original post that Woodsman may have tried or looked at vnstat already. I will bow out to wash the egg from my face.

are you looking for something like iptraf? it's in the "n" section of standard slackware. :)

Here's the situation, to try to minimize the noise... OP has multiple machines on a LAN. Traffic moves between the machines as well as to/from the machines and the Internet. The OP is interested in the traffic to/from the Internet, but not the traffic from machine to machine on the LAN. Since all the traffic is going through the same interface (e.g. eth0), the required software must be able to discriminate based on source and destination IP, and only aggregate traffic who source OR destination are not LAN IPs (see the wireshark filter example in #6).

Very old (respectively speaking.) More importantly, the tool does not recognize a bridge interface (br0). I'm using VirtualBox 2.0.8 and I create a bridge network interface to run my virtual machines on a host network. Possibly the newer versions of VirtualBox do not require that setup, but as my office machine is stable, I prefer not to fiddle with updating VirtualBox right now.

Yup, that is a good summary. :)

If I was running a router/gateway from a Slackware box, I then would have two network cards. One for the LAN and one for the ISP. I then could distinguish the traffic type.

I did read a little about using iptables to track traffic. I have not dug deeper into that idea. A disadvantage with that idea is I would have to configure every box with the same setup and then merge data across my LAN. That solution also does not help me with any Windows machine accessing the internet, unless I force those machines through a proxy running the traffic monitoring iptables rules.

I think my best solution is grabbing the ifconfig or /proc/net/dev information from the router. I have jffs configured in the router so I would have a place to permanently store the information. I would have to write a shell script to parse the data by day. Probably add two cron jobs, one to grab the data at, say, one minute intervals; and the second to parse the data correctly across midnight. I think a simple text file similar but not exactly the same as the KNemo data files will suffice. I could have my primary office machine, which is on almost every day, grab the data for easier local viewing.

What do you think?