LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-26-2010, 07:11 AM   #1
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Rep: Reputation: 0
Question search a string in logs of last 15 minutes from current time.


I want to run a script that runs after every 15 minutes that i will do using crontab.
But in script want to search a string from the last 15 minutes logs in log file containing data of whole day.

Please guide me how i can search the string according to time difference that is logs from from current time and current time - 15 minutes.

Sample logs are as follows:

26-Aug-2010 16:38:46,055|9172310750|subscription_app|31ba267e%3A12aadd47bdc%3A50e9|ChargeAmount|ChargingIntercep tor - subscriber details processed sucessfully- {arg0.referenceCode=balanceEnquiry:true;subsChannel:Unknown;channelType:Subscription;transactionId:3 1ba267e%3A12aadd47bdc%3A50e9;pricePtAvl:true;eventType:subscription;contentId:4945;serviceId:CR03;Ci rcle_Name:MU;Circle_ID:22;isRated:Yes;productName:VAS0003ALL;basePrice:0.0;subsType:RECURRING;Sub_Pr ofile:Pre-Paid, arg0.endUserIdentifier=9172310750, arg0.charge.description= Retrieve-Balance , arg0.charge.currency=INR, arg0.charge.code=, arg0.charge.amount=0.0}
26-Aug-2010 16:38:46,056|9172310750|subscription_app|31ba267e%3A12aadd47bdc%3A50e9|ChargeAmount|CustomCDRInterce ptor - CDR Info[Optional_Field1:,Subscription_Channel:Unknown,Optional_Field2:,Transaction_ID:,Content_ID:4945,IMEI: ,Product_Name:VAS0003ALL,PPL_FLAG:,Charge_Code:,Base_Price:0.0,CustomerID:B_16959389,Circle_Name:MU, Sender_MSISDN:,IMSI:405926120251895,Content_Status:,Location:MU,Circle_ID:22,Original_Content_Owner_ ID:,CPNAME:default_provider,Content_Price:0.0,Zoneefault,Content_Name:,Static_ID:MU#15140988,Exter nal_Correlation_Id:31ba267e%3A12aadd47bdc%3A50e9,Subscription_Type:RECURRING,MSISDN:9172310750,Trans action_Mode:Subscription,Transaction_DateTime:2010-08-26 16:38:46 GMT+05:30,Content_Type:,Sub_Profile:Pre-Paid,CPID:,Other_Info:]
26-Aug-2010 16:38:46,057|9172310750|subscription_app|31ba267e%3A12aadd47bdc%3A50e9|ChargeAmount|GetBalance|Payme ntPlugin-Request - Get User Balance of: 9172310750
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 08-26-2010, 07:31 AM   #2
premkumarg
LQ Newbie
 
Registered: Aug 2010
Posts: 4

Rep: Reputation: 0
RE: search a string in logs of last 15 minutes from current time

You can try sed command
sed -n '/<xyz>/p' $filename

<xyz> - represent the text to search in the $filename

You can use your own pattern to search the required string.

Last edited by premkumarg; 08-26-2010 at 07:34 AM.
 
Old 08-26-2010, 09:37 AM   #3
berbae
Member
 
Registered: Jul 2005
Location: France
Distribution: Arch Linux
Posts: 540

Rep: Reputation: Disabled
An idea to prevent you from manipulating time strings and from making calculations on them :

1) Memorize the last entry in the log file at time 0

2) 15 minutes later, search the string you want, but restricting the search from the previously memorized last entry to the end of file

3) Memorize the new last entry for the next call

example :
sed -n '/last entry memorized string/,$ p' logfilename|grep "string you want"

In the last entry memorized string, the special characters used in a regular expression are to be backslashed for them to be interpreted literally, ie change [] to \[\] in the string...

For that to work each entry has to be written on one line.
If this is not the case give more infos on the exact structure of the log file.

Edit
It is even simpler to work with line number :

1) Memorize the last line number of the last entry with
wc -l logfilename|cut -d ' ' -f1 >afilesomewhere

2) 15 minutes later your script uses :
lastline=$(<afilesomewhere)
sed -n "$lastline,$ p" logfilename|grep "string you want"

3) Replace the last line number :
wc -l logfilename|cut -d ' ' -f1 >afilesomewhere

Last edited by berbae; 08-26-2010 at 04:24 PM.
 
Old 08-27-2010, 12:35 AM   #4
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Question search a string in logs of last 15 minutes from current time

Hi Berbae,
Thanks for help.
But in logfile logs are continously appended as these are logs of currently running application.

Please suggest.

My Lofile structure is as follows:

27-Aug-2010 10:04:30,598|7396093482|subscription_app|-30d7c1db%3A12ab0529195%3A11f3|ChargeAmount|REInterceptor - Is already Rated [Yes] RatedPrice [0.0]
27-Aug-2010 10:04:30,598|7396093482|subscription_app|-30d7c1db%3A12ab0529195%3A11f3|ChargeAmount|ChargingInterceptor - subscriber details processed sucessfully- {arg0.referenceCode=balanceEnquiry:true;subsChannel:Unknown;channelType:Subscription;transactionId:-30d7c1db%3A12ab0529195%3A11f3;pricePtAvl:true;eventType:subscription;contentId:4945;serviceId:CR03;C ircle_Name:AP;Circle_ID:2;isRated:Yes;productName:VAS0003ALL;basePrice:0.0;subsType:RECURRING;Sub_Pr ofile:Pre-Paid, arg0.endUserIdentifier=7396093482, arg0.charge.description= Retrieve-Balance , arg0.charge.currency=INR, arg0.charge.code=, arg0.charge.amount=0.0}
27-Aug-2010 10:04:30,599|7396093482|subscription_app|-30d7c1db%3A12ab0529195%3A11f3|ChargeAmount|CustomCDRInterceptor - CDR Info[Optional_Field1:,Subscription_Channel:Unknown,Optional_Field2:,Transaction_ID:,Content_ID:4945,IMEI: ,Product_Name:VAS0003ALL,PPL_FLAG:,Charge_Code:,Base_Price:0.0,CustomerID:B_18150762,Circle_Name:AP, Sender_MSISDN:,IMSI:405819121068875,Content_Status:,Location:AP,Circle_ID:2,Original_Content_Owner_I D:,CPNAME:default_provider,Content_Price:0.0,Zone:,Content_Name:,Static_ID:AP#16942965,External_Corr elation_Id:-30d7c1db%3A12ab0529195%3A11f3,Subscription_Type:RECURRING,MSISDN:7396093482,Transaction_Mode:Subscri ption,Transaction_DateTime:2010-08-27 10:04:30 GMT+05:30,Content_Type:,Sub_Profile:Pre-Paid,CPID:,Other_Info:]
27-Aug-2010 10:04:30,600|7396093482|subscription_app|-30d7c1db%3A12ab0529195%3A11f3|ChargeAmount|GetBalance|PaymentPlugin-Request - Get User Balance of: 7396093482
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 10:04:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 10:04:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
27-Aug-2010 10:04:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL to be called : http://172.30.24.52/unitech_sms/unit...018&ContentID=
27-Aug-2010 10:04:30,755|9062342952|subscription_app|-7ff2f433%3A12ab1cd4b07%3A-7b6b|ChargeAmount|GetBalance|PaymentPlugin-Response - Retrieved Balance Bucket: 1;20091003;20110218;22.00;|
27-Aug-2010 10:04:30,756|9062342952|subscription_app|-7ff2f433%3A12ab1cd4b07%3A-7b6b|ChargeAmount|Exception: [Message:null]



Quote:
Originally Posted by berbae View Post
An idea to prevent you from manipulating time strings and from making calculations on them :

1) Memorize the last entry in the log file at time 0

2) 15 minutes later, search the string you want, but restricting the search from the previously memorized last entry to the end of file

3) Memorize the new last entry for the next call

example :
sed -n '/last entry memorized string/,$ p' logfilename|grep "string you want"

In the last entry memorized string, the special characters used in a regular expression are to be backslashed for them to be interpreted literally, ie change [] to \[\] in the string...

For that to work each entry has to be written on one line.
If this is not the case give more infos on the exact structure of the log file.

Edit
It is even simpler to work with line number :

1) Memorize the last line number of the last entry with
wc -l logfilename|cut -d ' ' -f1 >afilesomewhere

2) 15 minutes later your script uses :
lastline=$(<afilesomewhere)
sed -n "$lastline,$ p" logfilename|grep "string you want"

3) Replace the last line number :
wc -l logfilename|cut -d ' ' -f1 >afilesomewhere
 
Old 08-27-2010, 12:39 AM   #5
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Question search a string in logs of last 15 minutes from current time

Dear Prem KumarG,
There is no time check in your post.
Here my concern is totally focused on time only.
Please suggest according to time slice of 15 minutes.


Quote:
Originally Posted by premkumarg View Post
You can try sed command
sed -n '/<xyz>/p' $filename

<xyz> - represent the text to search in the $filename

You can use your own pattern to search the required string.
 
Old 08-27-2010, 05:40 AM   #6
quanta
Member
 
Registered: Aug 2007
Location: Vietnam
Distribution: RedHat based, Debian based, Slackware, Gentoo
Posts: 724

Rep: Reputation: 100Reputation: 100
Hi,

With the follow input:
Code:
$ cat grep_time_range.input 
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
Try this:
Code:
$ cat grep_time_range.sh 
#!/bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from
awk '$0>=from && $0<=to' from="$from" to="$to" grep_time_range.input
And result:
Code:
$ ./grep_time_range.sh 
27-Aug-2010 16:35:57
27-Aug-2010 16:20:57
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
PS: Remember to remove two 'echo' lines.
 
2 members found this post helpful.
Old 08-27-2010, 06:33 AM   #7
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Smile Thanks

Hi Quanta

Thanks, It really work..

Quote:
Originally Posted by quanta View Post
Hi,

With the follow input:
Code:
$ cat grep_time_range.input 
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
Try this:
Code:
$ cat grep_time_range.sh 
#!/bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from
awk '$0>=from && $0<=to' from="$from" to="$to" grep_time_range.input
And result:
Code:
$ ./grep_time_range.sh 
27-Aug-2010 16:35:57
27-Aug-2010 16:20:57
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018
PS: Remember to remove two 'echo' lines.
 
  


Reply

Tags
asap


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
search string in text and print string wolfipa Linux - Software 4 09-17-2009 09:50 AM
Search for a string manjeshjk Linux - Newbie 4 07-14-2009 02:46 AM
PHP: formatting current date & time as: 2006-04-12 13:47:36 and substract 15 minutes NaCo Linux - Software 4 11-13-2008 08:31 PM
Can you make search ...search a string in a link....a url...a web address aus9 LQ Suggestions & Feedback 4 04-16-2008 10:37 AM
C....Search a string for a string Scrag Programming 4 06-14-2004 05:15 PM


All times are GMT -5. The time now is 08:22 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration