Grep string from logs of last 1 hour on files of 2 different servers+calculate count

saurabhmehan · 09-03-2010, 02:20 AM

Hi,
I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files.

File structure is same on both servers and for reference as follows:

Code:

27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018

evo2 · 09-03-2010, 03:09 AM

The "last 1 hour" (ie 60 minutes) makes it non trivial.
If you are happy to just have within the current hour you could do something like the following where I'm assuming you are searching for "TransactionStatus:Requested"

Also, I'm taking your request for "grep" literally... I'd do this in python ;-)

Code:

#!/bin/bash
t=0
for s in server1 server2 ; do
   n=$(ssh $s "grep -cE ^\"$(date +'%d-%b-%Y %H).*TransactionStatus:Requested\" /var/log/foo.log")
   echo "$s, $n occurrences"
   t=$((t+n))
done
echo "Total $t occurrences"

Cheers,

Evo2.

tracertong · 09-03-2010, 03:43 AM

Assuming you have fuzzy date/time logic supported, a relative time can be calculated by simply:

Code:

user@server:~ $ date +'%d-%b-%Y %H:%M:%S' --date "1 hour ago"
03-Sep-2010 08:55:48

(This can be copared with the file date as a simple ASCII string.)

This a General Hint For Living, that not only makes for flawless logic - such that you no longer have to do any tedious mucking about, worrying about the middle of February (if you need to work out something like, this):

Code:

user@server:~ $ date --date "next month"
Sun Oct  3 09:57:52 BST 2010

...or:

Code:

user@server:~ $ date --date "fortnight ago"
Fri Aug 20 09:58:13 BST 2010

...but it also makes for much more human-readable logic. It is relatively easy to see what someone who says:

Code:

date --date "last Friday"
Fri Aug 27 00:00:00 BST 2010

... is trying to achieve.

One only has to have encountered code written by someone who A) tried to work it all out in long hand, and B) had to worry about what happened in the middle of February, to see how non-human-readable, some code, can become.

saurabhmehan · 09-03-2010, 04:27 AM

Hi,

@evo2 : I want this logic for last 15 minutes too.
Is there any idea for getting last 15 minutes logs and search for string count in last 15 minutes.

Quote:

Originally Posted by evo2

The "last 1 hour" (ie 60 minutes) makes it non trivial.
If you are happy to just have within the current hour you could do something like the following where I'm assuming you are searching for "TransactionStatus:Requested"

Also, I'm taking your request for "grep" literally... I'd do this in python ;-)

Code:

#!/bin/bash
t=0
for s in server1 server2 ; do
   n=$(ssh $s "grep -cE ^\"$(date +'%d-%b-%Y %H).*TransactionStatus:Requested\" /var/log/foo.log")
   echo "$s, $n occurrences"
   t=$((t+n))
done
echo "Total $t occurrences"

Cheers,

Evo2.

saurabhmehan · 09-03-2010, 05:10 AM

Find the code below which i have made but getting error.
The code checks the last 15 minutes logs and search the string and get the total count.Can anybody correct the following code or provide me the optimized way:

Code:

#!/bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh -n smehan@10.0.0.1 "awk '$0>=from && $0<=to' from=\"$from\" to=\"$to\" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep \"ShortCode=tel:${shortcode}\" | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
done

tracertong · 09-03-2010, 08:40 AM

Well, my own suggestion, was that the date, you have, is a standard, recognizable Unix time format - so you can really just extract that date, on the assumption that it will always be the first 20 characters of each line, and push it at the date command specifying the Unix Timestamp conversion:

Code:

date +"%s" --date "<feed-value>"

This obtains the Unix time stamp, as an int. Now do the same thing with "1 hour ago" (or "15 minutes ago", or whatever) and compare the two values: then do the extra work on whatever fits those criteria.

something like this:

Code:

#!/bin/bash

hourago=`date +"%s" --date "1 hour ago"`

IFS=$'\n'
for line in $(cat filename)
do
        date=`echo ${line} | cut -c 1-20`
        if [ `date +"%s" --date "$date"` -ge "$hourago" ]
        then
                echo "Your grep goes in here."
        fi
done

The advantage of this, is that the filter can be changed to anything you want:

Code:

filter=`date +"%s" --date "16 hours 12 minutes ago"`
filter=`date +"%s" --date "Next Tuesday week"
filter=`date +"%s" --date ""Fortnight last Wednesday"`

Whatever. They all parse as relative date values, and make what you are trying to achieve in the script more readable.

Once you have your matching lines, you can act on them how you want.

saurabhmehan · 09-06-2010, 12:22 AM

Check my output below and suggest the changes that have to be done or optimized way of doing that:

Code:

+ set -u
++ date '+%d-%b-%Y %T'
+ to='06-Sep-2010 10:46:22'
+ echo 06-Sep-2010 10:46:22
06-Sep-2010 10:46:22
++ date +%s
+ let from_in_seconds=1283750182-900
++ date -d @1283749282 '+%d-%b-%Y %T'
+ from='06-Sep-2010 10:31:22'
+ echo 06-Sep-2010 10:31:22
06-Sep-2010 10:31:22
+ shortcodes=("56882")
+ for shortcode in '${shortcodes[@]}'
+ count=0
++ awk '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
++ grep -c ShortCode=tel:56882
+ sh_count=0
++ expr 0 + 0
+ count=0
++ ssh -n smehan@172.30.16.225
+ grep -c ShortCode=tel:56882
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ sh_count2=awk
+ '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
./test_mo_count2: line 17: $0>=from && $0<=to: command not found
0
./test_mo_count2: line 18: sh_count2: unbound variable
+ count=
+ echo '56882 : '
56882 :