LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-27-2006, 01:13 PM   #1
altrob
LQ Newbie
 
Registered: Sep 2006
Posts: 3

Rep: Reputation: 0
Shell script help with function and output format


Greetings,

I'm new to writing shell scripts but after searching through this forum I've gotten most of what I want. However, I'm still having an issue. I am attempting to backload data from hundred's of large csv's into RRD. Essentially I want to parse out info from hundred's of columns in hundred's of files into 4 columns of data I want in one file. Essentially the date in unix time, the min, max, and average of a reading (say CPU) for the day, compiling all the days data into one file. The problem is I can't get my head around how to get the data to return on the same line.

data:

file1.csv
4 lines of headers I don't need
1-JUL-2006 17:41:00.00,111.42,xxx,yyy,zzz,etc..
1-JUL-2006 17:42:00.00,69.87,xxx,yyy,zzz,etc..
1-JUL-2006 17:43:00.00,101.10,xxx,yyy,zzz,etc..
etc...

file2.csv
4 lines of headers I don't need
2-JUL-2006 17:41:00.00,111.42,xxx,yyy,zzz,etc..
2-JUL-2006 17:42:00.00,69.87,xxx,yyy,zzz,etc..
2-JUL-2006 17:43:00.00,101.10,xxx,yyy,zzz,etc..

etc..

What I have is:
Code:
#!/bin/bash
destdir=/home/t4data/working
for f in $destdir/*.csv
        do
                tail -1 $f |cut -c1-12| while read line # gives me the date for the file
                        do
                                date -d "$line" +%s	# convert to unix time 
                        done
                tail -n+5 $f |awk -F, ' { print $7 } ' |awk -f sum.awk # min,max,ave
        done
where sum.awk is

Code:
$ cat sum.awk
{
    if (NR == 1) {
        sum=min=max=$1;
    } else {
        sum += $1;
        min = (min < $1) ? min : $1;
        max = (max > $1) ? max : $1;
    }
}
END {
#    print "The average is: " sum/NR " min: " min " max: " max;
    print " min: " min " max: " max " average: "sum/NR;
}
Output:

1153022400
min: 44.45 max: 666.22 average: 136.055
1153108800
min: 51.28 max: 1298.38 average: 374.275
etc...

I realise why the data returns on two separate lines, and I thought it would be simple to correct, but no matter what I try I can't seem to get to my desired format of:

1153022400 min: 44.45 max: 666.22 average: 136.055
1153108800 min: 51.28 max: 1298.38 average: 374.275
etc...

Thoughts?
 
Old 09-27-2006, 02:05 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,379

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
bit lost here, are you saying you're only looking at the very last line of each file? if so why is there a do loop there?

now presumably you simply want to get a timestamp to a variable, and then get the rrd data to a variable and then print them both?
Code:
#!/bin/bash
destdir=/home/t4data/working
for f in $destdir/*.csv
        do
                lastline=$(tail -1 $f |cut -c1-12)
                rrddate=$(date -f $lastline +%s)
                rrddata=$(tail -n+5 $f |awk -F, ' { print $7 } ' | awk -f sum.awk )
                echo $rrddate $rrddata
        done
i'd be thinking those tails and whatnot could do with looking at but i'm not in a position to suss that out without breaking it somewhere... but is that not the kind of result you're after?
 
Old 09-27-2006, 02:10 PM   #3
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371
Hi,

Just looking at what you asked:

Change this:

date -d "$line" +%s # convert to unix time

Into:

thisDate=`date -d "$line" +%s` # convert to unix time
echo -n $thisDate


Make it a two step process.
1) put date in a variable,
2) output this variable using echo -n (or a printf construct if you like that better).

The echo -n makes sure that there is no newline after it prints the date.

Hope this helps.
 
Old 09-27-2006, 02:39 PM   #4
altrob
LQ Newbie
 
Registered: Sep 2006
Posts: 3

Original Poster
Rep: Reputation: 0
Smile

Quote:
Originally Posted by acid_kewpie
bit lost here, are you saying you're only looking at the very last line of each file? if so why is there a do loop there?
I was only looking at the last line of each file for the date parameter. Each file had hundreds of lines of data, but each line of a particular file has the same date. Parse a file down to one line for date, min, max, ave for the day, move to next file, repeat.
Code:
#!/bin/bash
destdir=/home/t4data/working
for f in $destdir/*.csv
        do
                lastline=$(tail -1 $f |cut -c1-12)
                rrddate=$(date -f $lastline +%s)
                rrddata=$(tail -n+5 $f |awk -F, ' { print $7 } ' | awk -f sum.awk )
                echo $rrddate $rrddata
        done
I had previously done various forms of what you suggested ( I only posted my most recent attempt), but each one, including your code resulted in a newline ie

1153022400
min: 44.45 max: 666.22 average: 136.055
1153108800
min: 51.28 max: 1298.38 average: 374.275

Quote:
Originally Posted by druuna

thisDate=`date -d "$line" +%s` # convert to unix time
echo -n $thisDate


Make it a two step process.
1) put date in a variable,
2) output this variable using echo -n (or a printf construct if you like that better).

The echo -n makes sure that there is no newline after it prints the date.

Hope this helps.
This is exactly what I needed. I tried working with echo -n without success, but I wasn't getting the variable set properly with the tilde (though I thought I tried that).

I works perfectly! Thanks very much to both of you for your help, especially druuna for providing the key that I was missing.

cheers!

Last edited by altrob; 09-27-2006 at 02:40 PM.
 
Old 09-27-2006, 02:51 PM   #5
altrob
LQ Newbie
 
Registered: Sep 2006
Posts: 3

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by altrob
Code:
#!/bin/bash
destdir=/home/t4data/working
for f in $destdir/*.csv
        do
                lastline=$(tail -1 $f |cut -c1-12)
                rrddate=$(date -f $lastline +%s)
                rrddata=$(tail -n+5 $f |awk -F, ' { print $7 } ' | awk -f sum.awk )
                echo $rrddate $rrddata
        done
I had previously done various forms of what you suggested ( I only posted my most recent attempt), but each one, including your code resulted in a newline
Let me correct that. That does indeed work acid_kewpie if you change the date -f to date -d. The error from date -f introduced a newline when I quickly tried it. So you both gave me a working solution, both of which I was dancing around but obviously not getting it exactly as needed. Thanks to both for such quick responses!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
can a C function return value to Shell Script variable yarnar Programming 17 06-02-2010 05:54 PM
PHP function system() - format the output fatrandy13 Programming 5 09-20-2006 08:53 AM
insert lines in a function using shell script shyamdey Programming 1 08-30-2006 07:48 AM
How to change parameter values of a function in shell script? Bassam Programming 0 01-25-2004 09:52 AM


All times are GMT -5. The time now is 03:50 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration