LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-03-2010, 03:20 AM   #1
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Rep: Reputation: 0
Question Grep string from logs of last 1 hour on files of 2 different servers+calculate count


Hi,
I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files.

File structure is same on both servers and for reference as follows:
Code:
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
 
Old 09-03-2010, 04:09 AM   #2
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
The "last 1 hour" (ie 60 minutes) makes it non trivial.
If you are happy to just have within the current hour you could do something like the following where I'm assuming you are searching for "TransactionStatus:Requested"

Also, I'm taking your request for "grep" literally... I'd do this in python ;-)
Code:
#!/bin/bash
t=0
for s in server1 server2 ; do
   n=$(ssh $s "grep -cE ^\"$(date +'%d-%b-%Y %H).*TransactionStatus:Requested\" /var/log/foo.log")
   echo "$s, $n occurrences"
   t=$((t+n))
done
echo "Total $t occurrences"
Cheers,

Evo2.
 
Old 09-03-2010, 04:43 AM   #3
tracertong
Member
 
Registered: Jun 2010
Posts: 34

Rep: Reputation: 16
Assuming you have fuzzy date/time logic supported, a relative time can be calculated by simply:
Code:
user@server:~ $ date +'%d-%b-%Y %H:%M:%S' --date "1 hour ago"
03-Sep-2010 08:55:48
(This can be copared with the file date as a simple ASCII string.)

This a General Hint For Living, that not only makes for flawless logic - such that you no longer have to do any tedious mucking about, worrying about the middle of February (if you need to work out something like, this):
Code:
user@server:~ $ date --date "next month"
Sun Oct  3 09:57:52 BST 2010
...or:
Code:
user@server:~ $ date --date "fortnight ago"
Fri Aug 20 09:58:13 BST 2010
...but it also makes for much more human-readable logic. It is relatively easy to see what someone who says:
Code:
date --date "last Friday"
Fri Aug 27 00:00:00 BST 2010
... is trying to achieve.

One only has to have encountered code written by someone who A) tried to work it all out in long hand, and B) had to worry about what happened in the middle of February, to see how non-human-readable, some code, can become.

Last edited by tracertong; 09-03-2010 at 07:00 AM.
 
Old 09-03-2010, 05:27 AM   #4
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Question Grep string from logs of last 1 hour on files of 2 different servers+calculate count

Hi,

@evo2 : I want this logic for last 15 minutes too.
Is there any idea for getting last 15 minutes logs and search for string count in last 15 minutes.


Quote:
Originally Posted by evo2 View Post
The "last 1 hour" (ie 60 minutes) makes it non trivial.
If you are happy to just have within the current hour you could do something like the following where I'm assuming you are searching for "TransactionStatus:Requested"

Also, I'm taking your request for "grep" literally... I'd do this in python ;-)
Code:
#!/bin/bash
t=0
for s in server1 server2 ; do
   n=$(ssh $s "grep -cE ^\"$(date +'%d-%b-%Y %H).*TransactionStatus:Requested\" /var/log/foo.log")
   echo "$s, $n occurrences"
   t=$((t+n))
done
echo "Total $t occurrences"
Cheers,

Evo2.
 
Old 09-03-2010, 06:10 AM   #5
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Find the code below which i have made but getting error.
The code checks the last 15 minutes logs and search the string and get the total count.Can anybody correct the following code or provide me the optimized way:
Code:
#!/bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh -n smehan@10.0.0.1 "awk '$0>=from && $0<=to' from=\"$from\" to=\"$to\" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep \"ShortCode=tel:${shortcode}\" | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
done
 
Old 09-03-2010, 09:40 AM   #6
tracertong
Member
 
Registered: Jun 2010
Posts: 34

Rep: Reputation: 16
Well, my own suggestion, was that the date, you have, is a standard, recognizable Unix time format - so you can really just extract that date, on the assumption that it will always be the first 20 characters of each line, and push it at the date command specifying the Unix Timestamp conversion:
Code:
date +"%s" --date "<feed-value>"
This obtains the Unix time stamp, as an int. Now do the same thing with "1 hour ago" (or "15 minutes ago", or whatever) and compare the two values: then do the extra work on whatever fits those criteria.

something like this:

Code:
#!/bin/bash

hourago=`date +"%s" --date "1 hour ago"`

IFS=$'\n'
for line in $(cat filename)
do
        date=`echo ${line} | cut -c 1-20`
        if [ `date +"%s" --date "$date"` -ge "$hourago" ]
        then
                echo "Your grep goes in here."
        fi
done
The advantage of this, is that the filter can be changed to anything you want:
Code:
filter=`date +"%s" --date "16 hours 12 minutes ago"`
filter=`date +"%s" --date "Next Tuesday week"
filter=`date +"%s" --date ""Fortnight last Wednesday"`
Whatever. They all parse as relative date values, and make what you are trying to achieve in the script more readable.

Once you have your matching lines, you can act on them how you want.

Last edited by tracertong; 09-03-2010 at 09:44 AM.
 
Old 09-06-2010, 01:22 AM   #7
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Check my output below and suggest the changes that have to be done or optimized way of doing that:

Code:
+ set -u
++ date '+%d-%b-%Y %T'
+ to='06-Sep-2010 10:46:22'
+ echo 06-Sep-2010 10:46:22
06-Sep-2010 10:46:22
++ date +%s
+ let from_in_seconds=1283750182-900
++ date -d @1283749282 '+%d-%b-%Y %T'
+ from='06-Sep-2010 10:31:22'
+ echo 06-Sep-2010 10:31:22
06-Sep-2010 10:31:22
+ shortcodes=("56882")
+ for shortcode in '${shortcodes[@]}'
+ count=0
++ awk '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
++ grep -c ShortCode=tel:56882
+ sh_count=0
++ expr 0 + 0
+ count=0
++ ssh -n smehan@172.30.16.225
+ grep -c ShortCode=tel:56882
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ sh_count2=awk
+ '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
./test_mo_count2: line 17: $0>=from && $0<=to: command not found
0
./test_mo_count2: line 18: sh_count2: unbound variable
+ count=
+ echo '56882 : '
56882 :
 
  


Reply

Tags
asap


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to grep a string from all the files under / thomas2004ch Linux - Newbie 23 09-16-2009 09:16 PM
Find/grep/wc command to find matching files, print filename and word count dbasch Linux - Newbie 10 09-14-2009 06:55 PM
grep string with space (2 word string) casperdaghost Linux - Newbie 7 08-24-2009 03:11 AM
read files from a folder and grep a string bhagirathi Programming 6 07-06-2009 07:27 AM
My servers go down every weekend at fixed hour. Why? adrian.carciumaru Linux - Newbie 10 09-30-2008 04:35 AM


All times are GMT -5. The time now is 09:48 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration