LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-05-2010, 09:34 AM   #1
kcleveland
LQ Newbie
 
Registered: Jun 2010
Posts: 28

Rep: Reputation: 0
Question Shell script network data collection anomaly- need help for workarounds/changes


Hey everyone-

This is my first post to these forums (well, aside from the intro post )! Looking forward to exchanging ideas with all of you!

I have a script that I am using to try to measure network throughput over the ethernet ports. This is at work in order to test the performance of our onboard ethernet ports (planar) versus our pci-e (and daughter) ethernet pluggable cards.

The script is very long (and calls other scripts), so I will try to only show the parts that seem to be producing this weird little bug:

Code:
#Robert de Bock's "A shell script to measure network
#throughput on Linux machines" served as the basis for
#this part of the script.  Thank you for your help Robert!

#the $interface here is eth0 by default, but it is set from prompt args

printTXbytes(){
/sbin/ifconfig "$interface" | grep "TX bytes" | cut -d: -f3 | awk '{ print $1 }'
}

printRXbytes(){
/sbin/ifconfig "$interface" | grep "RX bytes" | cut -d: -f2 | awk '{ print $1 }'
}

bytestohumanreadable(){
multiplier="0"
number="$1"
while [ "$number" -ge 1024 ] ; do
  multiplier=$(($multiplier+1))
  number=$(($number/1024))
done
case "$multiplier" in
  1)
   echo "$number KB"
  ;;
  2)
   echo "$number MB"
  ;;
  3)
  echo "$number GB"
  ;;
  4)
  echo "$number TB"
  ;;
  *)
   echo "$1 B"
  ;;
esac
}

#below, usleep is also set by prompt args- default it is 1000000 usecs.
#counter is set by prompt arguments also, default is 60 iterations.

printResults (){
startTime=$(date +%s.%N)
while [ "$counter" -ge 0 ] ; do
  counter=$(($counter - 1))
  if [ "$rxbytes" ] ; then
   oldrxbytes="$rxbytes"
   oldtxbytes="$txbytes"
  fi
  rxbytes=$(printrxbytes)
  txbytes=$(printtxbytes)
  rxpackets=$(printrxpackets)
  txpackets=$(printtxpackets)
  ipaddr=$(printIP)
  cpuLoad=$(printCPULoad)
  memTotal=$(printTotalMem)
  	byteFilterOne=$((memTotal*1024))
  	memTotal=$byteFilterOne
  memUsed=$(printUsedMem)
	byteFilterTwo=$((memUsed*1024))
	memUsed=$byteFilterTwo
  if [ "$oldrxbytes" -a "$rxbytes" -a "$oldtxbytes" -a "$txbytes" ] ; then
   echo "================================================================="
   echo "Monitoring $interface @ $ipaddr every $usleep usec(s)."
   echo ""
   echo "$cpuLoad"
   echo ""
   echo "Total Memory ~ $(bytestohumanreadable $memTotal)  Used Memory ~ $(bytestohumanreadable $memUsed)" 
   echo "================================================================="
   echo ""
   echo "RXbytes = $(bytestohumanreadable $(($rxbytes - $oldrxbytes)))       TXbytes = $(bytestohumanreadable $(($txbytes - $oldtxbytes)))"
   	timeStamp=$(date +%M:%S)
   echo "$timeStamp $(($rxbytes - $oldrxbytes))" >> rxgrph.dat
   echo "$timeStamp $(($txbytes - $oldtxbytes))" >> txgrph.dat
   echo ""
   echo "RXpackets = $rxpackets       TXpackets = $txpackets"
   echo ""
   echo "--------------------------------------------------------------"
echo "RXbytes total = $(bytestohumanreadable $rxbytes)   TXbytes total = $(bytestohumanreadable $txbytes)"
   echo "--------------------------------------------------------------"
  fi
  usleep "$usleep"
  clear
done
What we're concerned with is what is getting put into the rxgrph.dat and txgrph.dat files. It seems that no matter what, there is usually at least 1 entry (no more than 3) into the .dat file that is double the amount of all of the other entries, causing a strange anomalous spike when it is graphed. A snapshot of a portion of the rxgrph.dat file:

Code:
29:36 9519552
29:37 9519616
29:38 19039552
29:39 9519680
29:40 9519552
First I started tinkering with the rate of the loop versus the rate that ifconfig is capable of refreshing at- which near as I can tell is 1 second. Also, the timestamps in the rxgrph.dat file seem to indicate that the sampling is indeed being done every second, but still the doubled number shows up. I had no success there, and now I'm thinking it could be caused by the way I have scripted something.

I'm still stuck on the idea that I'm catching ifconfig twice before it refreshes, hence the doubled number, but I can't really find any good info on the workings behind it to form some new ideas or workarounds. Suggestions on how to fix this anyone?

(let me know if I need to post more of the script)
Thanks in advance!

Last edited by kcleveland; 06-05-2010 at 09:35 AM.
 
Old 06-05-2010, 10:31 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
First of all welcome.

0) Why re-invent the wheel (and failures)? There's a gazillion Off-The-Shelf tools to use and customize, 1) '/sbin/ip -s link' is twice as fast as your '/sbin/ifconfig' line, 2) If you feel like using ifconfig then why not 'VALUE=$(/sbin/ifconfig eth0 | awk '/TX bytes/ {print $2}'); VALUE=${VALUE#bytes:}'? 3) No, even better: skip the whole 'ifconfig' part as it requires you to start the app each time while you can awk or grep the value from /proc/net/dev! 4) The best way to troubleshoot a script and understand what is happening is debugging it (as in '#!/bin/bash -vx') and reading back the output. Should be easy to see if its wrong input values, variables not being reset or Something Completely Different.
 
Old 06-05-2010, 02:35 PM   #3
kcleveland
LQ Newbie
 
Registered: Jun 2010
Posts: 28

Original Poster
Rep: Reputation: 0
Thank you for the reply! You have given me several new ideas to contemplate. I appreciate your input and will look into getting the data from /sbin/iplink or /proc/net/dev instead- definitely sounds like a better solution. Must say I should I have researched those before we started.
 
Old 06-07-2010, 09:09 AM   #4
kcleveland
LQ Newbie
 
Registered: Jun 2010
Posts: 28

Original Poster
Rep: Reputation: 0
I finally got a chance to try getting the data from /proc/net/dev and it works like a charm- the graphs are pretty and smooth now

A side note- I also managed to find a short blurb about how these utilities in the linux 2.4 kernel use a timer of about 0.9765 seconds - in the linux 2.6 kernel it is a 1 second timer. Definitely a good bit of info to know.

Thanks for your help!
 
Old 06-07-2010, 10:35 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
You're welcome.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Gathering data from different servers via Shell Script janak_17 Linux - Newbie 11 07-17-2009 04:01 AM
Ext4 data loss; explanations and workarounds H_TeXMeX_H Linux - News 11 03-27-2009 04:16 PM
shell script to find data that's all in one line wattz Programming 5 07-10-2008 04:48 PM
How to read data from file to use in shell script? ozymandias Linux - Newbie 7 10-27-2006 01:19 PM
Home Office Biotech Data Mining - Data Collection Adler Linux - General 20 11-03-2004 04:17 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration