Reading a .CSV file and then calculating average per minute basis in shell script.

krishdeeps · 04-23-2010, 04:19 PM

Hello,

I am new to shell script and to this form as well, I did try to search for a similar post like mine here, but could not find one.

I need some help and here is what I'm trying to do:

I am trying to grep server logs to find a specific string and then capture the time stamp and the value of that grep string in them. The log file prints out messages on per sec basis.

My script is able to grep the server logs for the entire period of my load runs and then outputted it to a .csv file too.
Unfortunately this .csv file is too large to extract it on my PC and to generate graphs as it exceeds the excel limit. I need some help on how to read this .csv file in a shell script and then take an average on per min basis before I can export it out on my desktop and generate graphs for analysis.

Any help would be greatly appreciated. Thanks.

example of of the out in my .csv file:

4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00 Total 355
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00 Total 474
4/19/2010 19:00 Total 494
4/19/2010 19:00
4/19/2010 19:00 Total 447
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 173
4/19/2010 19:01 Total 562
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 451
4/19/2010 19:01 Total 920
4/19/2010 19:01 Total 290
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 268
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 335
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 854
4/19/2010 19:01
4/19/2010 19:01 Total 1067
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 680
4/19/2010 19:01 Total 558
4/19/2010 19:01

pixellany · 04-23-2010, 04:38 PM

Out of time at the moment---but here is my hunch of how to do this:

This is only pseudocode and may still have some bugs.....

Code:

initialize the time stamp  (this requires reading the first line only---I'm assuming that "while read" will still start at the beginning.)
while read; do
   initialize variable to zero
   test current line to see if time stamp has changed---
       no---accumulate the total value into  variable
            continue the loop (ie branch to "while")
       yes---output the time stamp and the variable to a file  (see later)
done (branch to while automatically) <your_data_file > your_output_file

You should wind up with a file looking like this:

Code:

time stamp 1         total
time stamp 2         total

etc.