[SOLVED] awk to parse time and file transfer size

jzoudavy · 10-13-2015, 02:11 PM

It works, but I still get these weird operand errors and it still outputs a "total" message at the end.

Code:

./ftp_analyzer_grail.sh
Oct 7 13 total 5867354816 bytes
Oct 6 09 total 2340897 bytes
./ftp_analyzer_grail.sh: line 39: DL_SIZE +=  : syntax error: operand expected (error token is "+=  ")
./ftp_analyzer_grail.sh: line 40: DL_RATE +=  : syntax error: operand expected (error token is "+=  ")
   total 0 bytes

grail · 10-13-2015, 02:40 PM

I think you must have blank lines in your input file, as the errors are saying that there is nothing to add to our variables and hence the zero total at the end.
Try throwing this in as the first line inside the while loop:

Code:

[[ "$line" ]] || continue

I thought I would also an awk version I threw together as well:

Code:

#!/usr/bin/awk -f

{
  month = $1
  day = $2
  hour = $3
  rate = $6

  mon_sum[month] += rate
  day_sum[month][day] += rate
  hour_sum[month][day][hour] += rate
}

END{
  for( m in mon_sum)
  {
    print "Totals for the month of " m ":" 

    for( d in day_sum[m] )
    {   
      print " Hourly totals for day " d ":" 

      for( h in hour_sum[m][d] )
        print "  Hour :- " h " has a hourly sum of :- "hour_sum[m][d][h]

      print " Daily total is :- " day_sum[m][d]
    }   

    print "Monthly total is :- " mon_sum[m]
  }
}

You run this like so:

Code:

./script.awk input_file

jzoudavy · 10-13-2015, 02:59 PM

what would have been your perfered way to tackle this problem?

jzoudavy · 10-13-2015, 05:14 PM

Hi grail

Just want to say thank you for all the time and effort you have put in for helping me out with this. I thought I would share the final complete solution.

Code:

for use with vsftpd, enable dual mode logging, 

cd /var/log
awk '$10 ~/DOWNLOAD/ {print $2,$3,$4,$14,$15,$16}' vsftpd.log* > simplified.vsftpd.log
sed -i 's/:/ /g' simplified.vsftpd.log
sed -i 's/Kbyte/ Kbyte/g' simplified.vsftpd.log
sed -i 's/\./ /g' simplified.vsftpd.log
./ftp_analyzer.sh > ftp_util.txt

grail · 10-14-2015, 04:49 AM

Well the obvious comment would be that all the awk and seds could be completed inside the script

jzoudavy · 10-14-2015, 09:18 AM

Is there a way to put "./ftp_analyzer.sh > ftp_util.txt" inside the script as well?

grail · 10-14-2015, 09:38 AM

Yes, just setup the redirection to the file at the start of the script and all output will go there instead