parse file with awk and sum totals

master-of-puppets · 09-28-2014, 07:25 PM

Quote:

Originally Posted by pan64

So this file is generated with a perl script. I would suggest you to sum up those things in that perl script. In perl you can print the required result easily there is a built-in sort function. (so print what you want means reverse sort list and print the ordered result)

Hi Daniel,

I removed the sed statement out of the script that removed the string from the output but I still got the same results. Looking at jm_out does it still not have tabs? I also attached the resultant file jm_temp. Thanks for all your help.

Brian

danielbmartin · 09-28-2014, 07:58 PM

For the InFile which lacks tab characters, try this code ...

Code:

awk -F "" '{if ($1!=" ") {server=$0; next}
            if ($NF==":") {workspace=$0; next}
      print server,workspace,$0}' $InFile5 >$OutFile

Daniel B. Martin

master-of-puppets · 09-28-2014, 10:40 PM

Quote:

Originally Posted by danielbmartin

For the InFile which lacks tab characters, try this code ...

Code:

awk -F "" '{if ($1!=" ") {server=$0; next}
            if ($NF==":") {workspace=$0; next}
      print server,workspace,$0}' $InFile5 >$OutFile

Daniel B. Martin

Sadly it hangs and doesn't write anything to jm_temp. But hey thank you so much for your time. I think I'll just write a script that does everything including what the Perl script is now doing so I don't have to deal with this crazy output file. Thanks Daniel.

danielbmartin · 09-29-2014, 07:09 AM

Quote:

Originally Posted by master-of-puppets

Sadly it hangs and doesn't write anything to jm_temp.

This is puzzling because I tested with the large real-world InFile which you provided. Ran fine, with subsecond elapsed time, wrote 940 lines to the OutFile.

"Hangs and doesn't write anything" could be the program saying "Okay, I'm running, and waiting for input." Suggestion: check the way you specified the InFile. Keying error?

Daniel B. Martin

grail · 09-29-2014, 09:08 AM

As a side not on possible keying errors, please note the space between -F and the quote "" as if it is not there you will get the issue you have mentioned.

master-of-puppets · 09-30-2014, 01:06 AM

Quote:

Originally Posted by danielbmartin

This is puzzling because I tested with the large real-world InFile which you provided. Ran fine, with subsecond elapsed time, wrote 940 lines to the OutFile.

"Hangs and doesn't write anything" could be the program saying "Okay, I'm running, and waiting for input." Suggestion: check the way you specified the InFile. Keying error?

Daniel B. Martin

Hi Daniel,

The input file that we were working from was from this person who no longer works for the company and I didn't like the format of his output. It wasn't very useful. I created a new script that runs on each of the four build servers. It looks in local files only so as not to create any unnecessary network traffic. When each of the four servers finish generating their reports the files are then concatenated together and added to a weekly report along with some other data that was generated by my other script. The report is a csv file so that our admin can take it and generate pivot tables and graphs out of it.

Here's the new script that I came up with:

Code:

#!/bin/bash
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
case "$HOSTNAME" in
        sideshow) WORKSPACES=(bob mel sideshow-ws2) ;;
        simpsons) WORKSPACES=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
        moes)     WORKSPACES=(barney carl lenny moes-ws2) ;;
        flanders) WORKSPACES=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME_top_5_workspace.csv ]; then
echo "Top 5 consumers of space per workspace on server `hostname` $TODAY" > $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
echo ",,," >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
echo ",,," >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
for v in "${WORKSPACES[@]}"
do
echo "Top 5 consumers on workspace $v" >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
echo ",,," >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
find $BASE/$v -printf "%u  %s\n" | awk '{user[$1]+=$2}; END{ for( i in user) print i " " user[i]}' | sed -e '/^[0-9]/d' | sed -e 's/root//g' | sed -e '/^ [0-9]/d' | awk '{print $1, $2/1024/1024/1024, "GB"}' | sort -n -r -k2 | head -5 >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
echo ",,," >> $OUTPUT_DIR/"$HOSTNAME"_top_5_per_workspace.csv
done
fi

Sample output from host sideshow:

Code:

Top 5 consumers of space per workspace on server sideshow 09-29-14
,,,
,,,
Top 5 consumers on workspace bob
,,,
radickj 97.0708 GB
nichols2 90.4442 GB
sherryr 75.3845 GB
rabii 67.4304 GB
lefevre 39.0694 GB
,,,
Top 5 consumers on workspace mel
,,,
akrishna 125.225 GB
somyalip 124.585 GB
mvijayas 105.741 GB
release 102.279 GB
vuhang 83.4457 GB
,,,
Top 5 consumers on workspace sideshow-ws2
,,,
marlette 124.913 GB
iyershan 35.785 GB
starkd 19.3732 GB
jcook 3.8147e-06 GB
baylisn 3.8147e-06 GB
,,,

I want to thank you Daniel and everyone here. On a rivaling site they banned me for no good reason. They tried to contend that I was trying to trick the people on there into writing my scripts for me. None of the people on here have ever been condescending or judgmental like on that site. Thanks again.

Brian