LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 04-23-2010, 04:19 PM   #1
krishdeeps
LQ Newbie
 
Registered: Apr 2010
Posts: 1

Rep: Reputation: 0
Unhappy Reading a .CSV file and then calculating average per minute basis in shell script.


Hello,

I am new to shell script and to this form as well, I did try to search for a similar post like mine here, but could not find one.

I need some help and here is what I'm trying to do:

I am trying to grep server logs to find a specific string and then capture the time stamp and the value of that grep string in them. The log file prints out messages on per sec basis.

My script is able to grep the server logs for the entire period of my load runs and then outputted it to a .csv file too.
Unfortunately this .csv file is too large to extract it on my PC and to generate graphs as it exceeds the excel limit. I need some help on how to read this .csv file in a shell script and then take an average on per min basis before I can export it out on my desktop and generate graphs for analysis.

Any help would be greatly appreciated. Thanks.


example of of the out in my .csv file:

4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00 Total 355
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00 Total 474
4/19/2010 19:00 Total 494
4/19/2010 19:00
4/19/2010 19:00 Total 447
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:00
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 173
4/19/2010 19:01 Total 562
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 451
4/19/2010 19:01 Total 920
4/19/2010 19:01 Total 290
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 268
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 335
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 854
4/19/2010 19:01
4/19/2010 19:01 Total 1067
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01
4/19/2010 19:01 Total 680
4/19/2010 19:01 Total 558
4/19/2010 19:01
 
Old 04-23-2010, 04:38 PM   #2
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
Out of time at the moment---but here is my hunch of how to do this:

This is only pseudocode and may still have some bugs.....

Code:
initialize the time stamp  (this requires reading the first line only---I'm assuming that "while read" will still start at the beginning.)
while read; do
   initialize variable to zero
   test current line to see if time stamp has changed---
       no---accumulate the total value into  variable
            continue the loop (ie branch to "while")
       yes---output the time stamp and the variable to a file  (see later)
done (branch to while automatically) <your_data_file > your_output_file
You should wind up with a file looking like this:
Code:
time stamp 1         total
time stamp 2         total
etc.

Last edited by pixellany; 04-23-2010 at 04:43 PM.
 
  


Reply

Tags
shell script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script:- Reading numbers embedded in brackets from a text file rsan Linux - Newbie 6 07-05-2009 06:01 AM
Shell Script for CSV file comparision aravind_balan Programming 1 02-17-2009 03:33 AM
Calculate average from csv file in shell script khairilthegreat Linux - Newbie 5 11-21-2007 12:57 PM
Filtering a CSV file from web log with shell script? Micro420 Programming 8 08-22-2007 03:13 AM
Shell script to read from csv file hendemeg Programming 1 05-11-2004 08:23 PM


All times are GMT -5. The time now is 05:03 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration