[SOLVED] awk to parse time and file transfer size

chrism01 · 10-07-2015, 07:01 PM

Double [[ ]] are an improvement on single [ ] http://tldp.org/LDP/abs/html/testcon...ml#DBLBRACKETS .
Note that a pair of left or right square brackets count as a single token, so

Code:

# not like this
if [ [ $? -eq ] ] 

# but like this
if [[ $? -eq 0 ]]

Always use doubles instead of singles for square brackets.

Re (( )) - this is one of the ways to express performing an arithmetical op https://stackoverflow.com/questions/...s-curly-braces
NB: shell arithmetic is integer only - if you want fractional, use 'bc' eg

Code:

echo "scale=2;3/2"|bc
1.50

HTH

BTW, when debugging, try adding 'set -xv' as the 2nd line of your script; it shows what the parser is doing

jzoudavy · 10-07-2015, 07:46 PM

I actually tried that. But I just got an error about binary operator expected instead. I will need to double check again.

grail · 10-08-2015, 12:56 AM

Also, if you did wish to keep the floating point arithmetic, the awk solution can handle it.

AnanthaP · 10-08-2015, 01:02 AM

In your original file, the first 13 bytes (irrespective) give the day, month, date and hour. So that is the key.

I would use awk to build associative arrays of the sums of $8 and vomit it out on EOF. The core logic would look like:

Quote:

{
PK=substr($0,1,13);
sum[PK]+=$8
count[PK]++
}
END
{
for(i in sum) print i, sum[i], count[i]
}

OK

jzoudavy · 10-08-2015, 07:54 AM

Hi

thanks for pointing it out. I figured out why.

Code:

 

# should be like this
if [[ $? -eq 0 ]] && [[condition here]] && [[conditioner]]


previously i had this

if [ [] && [] ]

jzoudavy · 10-08-2015, 08:26 AM

Quote:

Originally Posted by grail

Also, if you did wish to keep the floating point arithmetic, the awk solution can handle it.

Hi grail

yes you are correct, the double round brackets makes it do arithmatic op.

funny how removing stuff makes you learn new things.

jzoudavy · 10-08-2015, 08:40 AM

ok, so now I got this, its producing some promising results but not what I was expecting.

Code:


#!/bin/bash
set -xv

MONTH=""
DAY=""
HOUR=""
DL_SIZE=0
DL_RATE=0

while read -a line; do
    CURR_MONTH=${line[1]}
	CURR_DAY=${line[2]}
    CURR_HOUR=${line[3]}
	CURR_DL_SIZE=${line[6]}
	CURR_DL_RATE=${line[8]}
	
    if [[ $CURR_MONTH == $MONTH ]] && [[ $CURR_DAY == $DAY ]] && [[ $CURR_HOUR == $HOUR ]]
	then
        $((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))
		echo "DL_SIZE is $DL_SIZE"
		$((DL_RATE=$DL_RATE+$CURR_DL_RATE))
		echo "DL_RATE is $DL_RATE"
    else # Update $MONTH and $DAY and $HOUR  
        if [[  $MONTH != "" ]]
		then
            echo "$MONTH $Day $HOUR total $DL_SIZE bytes \n" # introduce average DL rate later
        fi  
        MONTH=${line[1]}
		DAY=${line[2]}
		HOUR=${line[3]}
        
	fi  
done < simplified.vsftpd.log

# If we have reched EOF, print out the final month
echo "$MONTH $Day $HOUR total $DL_SIZE bytes "

Here is the output:

Code:

./ftp_analyzer.sh: line 19: 208431030: command not found
DL_SIZE is 208431030 \n
./ftp_analyzer.sh: line 21: 47604: command not found
DL_RATE is 47604 \n
./ftp_analyzer.sh: line 19: 416860806: command not found
DL_SIZE is 416860806 \n
./ftp_analyzer.sh: line 21: 125911: command not found
DL_RATE is 125911 \n
./ftp_analyzer.sh: line 19: 625289612: command not found
DL_SIZE is 625289612 \n
./ftp_analyzer.sh: line 21: 211343: command not found
DL_RATE is 211343 \n
./ftp_analyzer.sh: line 19: 833720399: command not found
DL_SIZE is 833720399 \n

... skipping some of the repetitive stuff here

./ftp_analyzer.sh: line 19: 5658923994: command not found
DL_SIZE is 5658923994 \n
./ftp_analyzer.sh: line 21: 765490: command not found
DL_RATE is 765490 \n
Oct  13 total 5658923994 bytes \n
Oct  09 total 5658923994 bytes

So the DL_SIZE is incrementing as it should, but the output is formated weird and those command not found is very strange and I am skipping the first DL_RATE for some reason.

grail · 10-08-2015, 09:07 AM

1. You do not need [[]] && [[]], you can simply place all the items inside a single set of brackets:

Code:

if [[ $CURR_MONTH == $MONTH && $CURR_DAY == $DAY && $CURR_HOUR == $HOUR ]]

2. $() will expand the commands inside and return the data. () on there own will either create an array or a sub-shell depending on the usage. So $(()) is of no particular use if you are not
assigning to a variable:

Code:

$((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))

# should be

((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))

# or

(( DL_SIZE = DL_SIZE + CURR_DL_SIZE ))

# or

(( DL_SIZE += CURR_DL_SIZE ))

The white space I have used is not required but I think it just looks clearer

Now once you fix up the above you are going to receive a new error and it is due to you placing the wrong items from the array into your variables

chrism01 · 10-08-2015, 08:16 PM

Also, it seems to me that really this line

Code:

    if [[ $CURR_MONTH == $MONTH ]] && [[ $CURR_DAY == $DAY ]] && [[ $CURR_HOUR == $HOUR ]]

should be using numeric comparison operators, not string ones; see http://www.tldp.org/LDP/abs/html/comparison-ops.html

jzoudavy · 10-09-2015, 03:19 PM

So I am beginning to think bash isn't the best way to tackle my problem. But I do believe I am close.

This is my output now. I am not sure how to fix the last bit, where instead of output the Oct 06 data it just prints Oct 7 again. I think it has to do with the way the loop is structured. by the time I should be printing the Oct 6th line it has already run to the next line.

Code:

# ./ftp_analyzer.sh
Oct 7 13 total 5867354816 bytes
Oct 7 13 total 0 bytes
Oct 7 13 total 0 bytes
   total 0 bytes

Below is what my code looks like now:

Code:

#!/bin/bash
#set -xv
#declare everything
MONTH=""
DAY=""
HOUR=""
DL_SIZE=0
DL_RATE=0
counter=0
isFirstLine=1
#if same month, same day and same hour, add the DL size and DL rate, DL rate for avg hourly transfer rate, which will be implemented later

while read -a line; do

        if [[ $isFirstLine -eq 1 ]]
        then # if this is the first line we are reading we want both current time and previous time to match
        CURR_MONTH=${line[0]}
        CURR_DAY=${line[1]}
        CURR_HOUR=${line[2]}

        MONTH=${line[0]}
        DAY=${line[1]}
        HOUR=${line[2]}
        isFirstLine=0
        else
        #if this is not the first line then we just need to read the new time
        CURR_MONTH=${line[0]}
        CURR_DAY=${line[1]}
        CURR_HOUR=${line[2]}

        fi


        CURR_DL_SIZE=${line[5]}
        CURR_DL_RATE=${line[7]}


    if [[ $CURR_MONTH -eq $MONTH && $CURR_DAY -eq $DAY &&  $CURR_HOUR -eq $HOUR ]]
        then
        ((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))
        ((DL_RATE=$DL_RATE+$CURR_DL_RATE))
        ((counter=$counter+1))
    else # Update the current $MONTH and $DAY and $HOUR
        # we have hit a new time, print result of old size

        echo "$MONTH $DAY $HOUR total $DL_SIZE bytes" # introduce average DL rate later

        #reset DL Size and Rate counter
        DL_SIZE=0
        DL_RATE=0
        counter=0

                CURR_MONTH=${line[0]}
                CURR_DAY=${line[1]}
                CURR_HOUR=${line[2]}


    fi
done < simplified.vsftpd.log


# If we have reched EOF, print out the final month
echo "$CURR_MONTH $CURR_DAY $CURR_HOUR total $DL_SIZE bytes"

grail · 10-10-2015, 01:33 AM

Ok, I am getting a little lost now with your logic, so we will start at the top:

1. Testing for the first line is rather pointless when you think that you can simply set MONTH, DAY and HOUR at the bottom of the loop and it will have the same affect

2. Even if you keep the test for the first line, there is no need to repeat the setting of the current items in both parts of the 'if'. Simply perform the test for true and set those items and then set the current items after the 'if'

3. In the 'else' portion of the next 'if' you set the current items for a third time. I do not see that this is needed at all

4. Why is the final echo using different variables than the previous one? On leaving the while loop all the variables will have been set and both current and standard items will be set to the same values

5. I assume we are currently ignoring the download rate and counter as they are not used or displayed

6. You must be using an alternate file format as download size is not in the sixth column in previous examples, hence your new script does not work for me.

jzoudavy · 10-13-2015, 01:09 PM

Hi Grail

Thanks for your comments. I have been running the logic through my head as well and you are right, the check first line wasn't necessary. It was because I noticed while debugging that the first line wasn't being included so I added that specific statement when in fact it is just a matter of updating the right variables.

And I have been doing a bit sanitizing of the input file as well so to simplify the scripting logic.

Code:

sample input:

Oct 7 13 36 31 208430822 bytes, 50790 63 Kbyte/sec
Oct 7 13 36 31 208431030 bytes, 47604 25 Kbyte/sec
Oct 7 13 36 33 208429776 bytes, 78307 48 Kbyte/sec
Oct 7 13 36 36 208428806 bytes, 85432 34 Kbyte/sec
Oct 7 13 36 39 208430787 bytes, 70624 04 Kbyte/sec
Oct 7 13 36 41 208429269 bytes, 19715 57 Kbyte/sec
Oct 7 13 36 42 208430626 bytes, 67402 46 Kbyte/sec
Oct 7 13 36 53 7004609 bytes, 55082 20 Kbyte/sec
Oct 7 13 36 53 7004596 bytes, 38641 45 Kbyte/sec
Oct 7 13 36 53 7004326 bytes, 53266 48 Kbyte/sec
Oct 7 13 36 53 7003780 bytes, 48976 23 Kbyte/sec
Oct 7 13 37 23 11188721 bytes, 57261 59 Kbyte/sec
Oct 7 13 37 23 11187409 bytes, 49023 38 Kbyte/sec
Oct 7 13 38 12 2013066706 bytes, 45416 61 Kbyte/sec
Oct 7 13 38 15 2344883553 bytes, 48741 71 Kbyte/sec
Oct 6 09 07 49 1170448 bytes, 28916 61 Kbyte/sec
Oct 6 09 07 49 1170448 bytes, 28916 61 Kbyte/sec

and here is the new code:

Code:

  #!/bin/bash
#need to target this per host
set -xv
#declare everything
MONTH=""
DAY=""
HOUR=""
CURR_MONTH=""
CURR_DAY=""
CURR_HOUR=""
DL_SIZE=0
DL_RATE=0
counter=0
 #if same month, same day and same hour, add the DL size and DL rate, DL rate for avg hourly transfer rate, which will be implemented later

while read -a line; do
 	MONTH=${line[0]}
    DAY=${line[1]}
    HOUR=${line[2]}
 	
	CURR_DL_SIZE=${line[5]}
    CURR_DL_RATE=${line[7]}
	#$((10#$hour)) is due to bash treats 0xxx as octo based not base 10
    if [[ $CURR_MONTH -eq $MONTH && $CURR_DAY -eq $DAY &&  $((10#$CURR_HOUR)) -eq $((10#$HOUR)) ]]
        then
        ((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))
        ((DL_RATE=$DL_RATE+$CURR_DL_RATE))
		((counter=$counter+1))
    else # Update the current $MONTH and $DAY and $HOUR
        # we have hit a new time, print result of old size
        echo "$CURR_MONTH $CURR_DAY $CURR_HOUR total $DL_SIZE bytes" # introduce average DL rate later
	     
        #reset DL Size and Rate counter
        DL_SIZE=0
        DL_RATE=0
		counter=0
		
		CURR_MONTH=${line[0]}
		CURR_DAY=${line[1]}
		CURR_HOUR=${line[2]}

    fi
	 
	
	
done < simplified.vsftpd.log

# If we have reched EOF, print out the final month
echo "$CURR_MONTH $CURR_DAY $CURR_HOUR total $DL_SIZE bytes"

and here is the output without debugging

Code:

   total 0 bytes
Oct 7 13 total 5658923994 bytes
Oct 6 09 total 1170448 bytes
   total 0 bytes

So currently we just have one problems: It skips the first line and last line and prints that "total" statement and I am not sure how to stop it. Below is the result with debugging.

Code:

+ read -a line
+ MONTH=Oct
+ DAY=7
+ HOUR=13
+ CURR_DL_SIZE=208430822
+ CURR_DL_RATE=50790
+ [[ '' -eq Oct ]]
+ [[ '' -eq 7 ]]
+ echo '   total 0 bytes'
   total 0 bytes
+ DL_SIZE=0
+ DL_RATE=0
+ counter=0
+ CURR_MONTH=Oct
+ CURR_DAY=7
+ CURR_HOUR=13
+ read -a line
+ MONTH=Oct
+ DAY=7
+ HOUR=13
+ CURR_DL_SIZE=208431030
+ CURR_DL_RATE=47604
+ [[ Oct -eq Oct ]]
+ [[ 7 -eq 7 ]]
+ [[ 13 -eq 13 ]]
+ (( DL_SIZE=0+208431030 ))
+ (( DL_RATE=0+47604 ))
+ (( counter=0+1 ))

On the bright side no more error statements

jzoudavy · 10-13-2015, 01:38 PM

i think i am close, all data from oct 7th now add correctly, but it won't do the rest of the dates for some reason.

jzoudavy · 10-13-2015, 01:51 PM

ok i got it. mostly.

sample input (note last line's DL size has been changed to allow for easier debugging):

Code:

[root@ssrarmftp2 log]# cat simplified.vsftpd.log
Oct 7 13 36 31 208430822 bytes, 50790 63 Kbyte/sec
Oct 7 13 36 31 208431030 bytes, 47604 25 Kbyte/sec
Oct 7 13 36 33 208429776 bytes, 78307 48 Kbyte/sec
Oct 7 13 36 36 208428806 bytes, 85432 34 Kbyte/sec
Oct 7 13 36 39 208430787 bytes, 70624 04 Kbyte/sec
Oct 7 13 36 41 208429269 bytes, 19715 57 Kbyte/sec
Oct 7 13 36 42 208430626 bytes, 67402 46 Kbyte/sec
Oct 7 13 36 53 7004609   bytes, 55082 20 Kbyte/sec
Oct 7 13 36 53 7004596   bytes, 38641 45 Kbyte/sec
Oct 7 13 36 53 7004326   bytes, 53266 48 Kbyte/sec
Oct 7 13 36 53 7003780   bytes, 48976 23 Kbyte/sec
Oct 7 13 37 23 11188721  bytes, 57261 59 Kbyte/sec
Oct 7 13 37 23 11187409  bytes, 49023 38 Kbyte/sec
Oct 7 13 38 12 2013066706 bytes, 45416 61 Kbyte/sec
Oct 7 13 38 15 2344883553 bytes, 48741 71 Kbyte/sec
Oct 6 09 07 49 1170448   bytes, 28916 61 Kbyte/sec
Oct 6 09 07 49 1170449   bytes, 28916 61 Kbyte/sec

Code:

#!/bin/bash
#need to target this per host
set -xv
#declare everything
MONTH=""
DAY=""
HOUR=""
CURR_MONTH=""
CURR_DAY=""
CURR_HOUR=""
DL_SIZE=0
DL_RATE=0
counter=0
isFirstLine=1
 #if same month, same day and same hour, add the DL size and DL rate, DL rate for avg hourly transfer rate, which will be implemented later

while read -a line; do


	if [[ $isFirstLine -eq 1 ]]
		then # if this is the first line we are reading we want both current time and previous time to match
		CURR_MONTH=${line[0]}
		CURR_DAY=${line[1]}
		CURR_HOUR=${line[2]}
	
		MONTH=${line[0]}
		DAY=${line[1]}
		HOUR=${line[2]}
		isFirstLine=0
	else
		#if this is not the first line then we just need to read the new time
		CURR_MONTH=${line[0]}
		CURR_DAY=${line[1]}
		CURR_HOUR=${line[2]}
	
	fi

  
 	
	CURR_DL_SIZE=${line[5]}
    CURR_DL_RATE=${line[7]}
	#$((10#$hour)) is due to bash treats 0xxx as octo based not base 10
    if [[ $CURR_MONTH -eq $MONTH && $CURR_DAY -eq $DAY &&  $((10#$CURR_HOUR)) -eq $((10#$HOUR)) ]]
        then
        ((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))
        ((DL_RATE=$DL_RATE+$CURR_DL_RATE))
		((counter=$counter+1))
    else # Update the current $MONTH and $DAY and $HOUR
        # we have hit a new time, print result of old size
        echo "$MONTH $DAY $HOUR total $DL_SIZE bytes over $counter downloads" # introduce average DL rate later
	     
        #reset DL Size and Rate counter
        DL_SIZE=0
        DL_RATE=0
		counter=0
		isFirstLine=1
	
		((DL_SIZE=$DL_SIZE+$CURR_DL_SIZE))
		((DL_RATE=$DL_RATE+$CURR_DL_RATE))
		((counter=$counter+1))
	fi
	 
	
	
done < simplified.vsftpd.log

output:

Code:

 
Oct 7 13 total 5867354816 bytes over 15 downloads
Oct 6 09 total 2340897 bytes over 2 downloads
./ftp_analyzer.sh: line 58: ((: DL_SIZE=0+: syntax error: operand expected (error token is "+")
./ftp_analyzer.sh: line 59: ((: DL_RATE=0+: syntax error: operand expected (error token is "+")

yeah I don't know why these two errors are happening. I think it is caused by hitting EOF or something, but everything else looks right

grail · 10-13-2015, 02:05 PM

Try this:

Code:

MONTH=""
DAY=""
HOUR=""
CURR_MONTH=
CURR_DAY=""
CURR_HOUR=""
DL_SIZE=0
DL_RATE=0
counter=0
#if same month, same day and same hour, add the DL size and DL rate, DL rate for avg hourly transfer rate, which will be implemented later

while read -a line; do
	MONTH=${line[0]}
	DAY=${line[1]}
	HOUR=${line[2]}

	#$((10#$hour)) is due to bash treats 0xxx as octo based not base 10
	if [[ "$CURR_MONTH" ]] && ( [[ "$CURR_MONTH" != "$MONTH" ]] || (( CURR_DAY != DAY || ${CURR_HOUR#0} != ${HOUR#0} )) )
	then
		# we have hit a new time, print result of old size
		echo "$CURR_MONTH $CURR_DAY $CURR_HOUR total $DL_SIZE bytes" # introduce average DL rate later

		#reset DL Size and Rate counter
		DL_SIZE=0
		DL_RATE=0
		counter=0
	fi

	CURR_MONTH=${line[0]}
	CURR_DAY=${line[1]}
	CURR_HOUR=${line[2]}

	(( DL_SIZE += ${line[5]} ))
	(( DL_RATE += ${line[7]} ))
	(( counter++ ))
done < simplified.vsftpd.log

# If we have reched EOF, print out the final month
echo "$CURR_MONTH $CURR_DAY $CURR_HOUR total $DL_SIZE bytes"