Latest LQ Deal: Linux Power User Bundle
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 09-08-2010, 03:49 AM   #1
Registered: Jul 2010
Posts: 44

Rep: Reputation: 0
Question Issue faced while accessing data from files on different servers.

I have two log files of same name and structure which resides on different servers with ssh-keygen installed.
I want to search for a list of strings available in an array on last 15 minutes logs of both files and calculate the total count of occurence of each string.

Sample format of both file is same and as follows:
08-Sep-2010 07:58:40,044|8923629811|aclwireless_app|1282498070897|NotifySmsReception|CP URL to be called :
08-Sep-2010 07:58:41,672|9126657443|aclwireless_app|1282498070897|NotifySmsReception|CP URL to be called :
08-Sep-2010 09:47:52,778|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|ChargingInterceptor - subscriber details processed sucessfully- {arg0.referenceCode=channelType:SMS;transactionId:1283918818906926;pricePtAvl:true;PPL_FLAG:FALSE;CorrelatorId:BEA1-611DC04C995F;contentType:Text;senderMSISDN:52211;contentId:1096;Subscriber_ID:BJ#12101396;SPACCOUNT_ID:PhoneyTunes;methodName:sendSms;Sub_Profile:Pre-Paid;IMSI:405876120820301;Customer_Retrieved:true;contentName:PT_JOKES;cpID:Phoneytunes;origContentOwnerId:10225;serviceId:PT_JOKES;Circle_Name:BJ;CustomerID:B_12318157;Circle_ID:4;isRated:No;MSISDN:9122796686;productName:JKGEN001;basePrice:-1.0, arg0.endUserIdentifier=9122796686, arg0.charge.description=CP is Calling TO This API, arg0.charge.currency=INR, arg0.charge.code=code0, arg0.charge.amount=0}
08-Sep-2010 09:47:52,789|7398965380|subscription_app|-7932740d%3A12aedfb6d15%3A-26d3|ChargeAmount|GetBalance|PaymentPlugin-Response -  Retrieved Balance Bucket: 1;20091003;20110306;122.50;|
08-Sep-2010 09:47:52,790|7398965380|subscription_app|-7932740d%3A12aedfb6d15%3A-26d3|ChargeAmount|Exception: [Message:null]
08-Sep-2010 09:47:52,810|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|DMGenerateLogInterceptor - InternalTransactionID:SDP-DM-31297182, ExternalTransactionID:1283918818906926, TransactionStatus:Sent For Delivery, INStatus:FAILED
08-Sep-2010 09:47:52,813|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|CustomCDRInterceptor - CDR Info[Optional_Field1:30.0,Subscription_Channel:,Optional_Field2:Subscribe,Transaction_ID:SDP-DM-31297182,Content_ID:1096,IMEI:,Product_Name:JKGEN001,PPL_FLAG:FALSE,Charge_Code:code0,Base_Price:0.0,CustomerID:B_12318157,Circle_Name:BJ,Sender_MSISDN:52211,IMSI:405876120820301,Content_Status:,Location:BJ,Circle_ID:4,Original_Content_Owner_ID:10225,CPNAME:PhoneyTunes,Content_Price:0,Zone:,Content_Name:PT_JOKES,Static_ID:BJ#12101396,External_Correlation_Id:1283918818906926,Subscription_Type:,MSISDN:9122796686,Transaction_Mode:SMS,Transaction_DateTime:2010-09-08 09:47:52 GMT+05:30,Content_Type:Text,Sub_Profile:Pre-Paid,CPID:Phoneytunes,Other_Info:]
08-Sep-2010 09:47:52,820|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|PaymentPlugin-Request -  [amountToBeCharged-0 | balanceChargeDetails0;0;0;0;| | transactionId-null | profileType-Pre-Paid ]
08-Sep-2010 09:47:52,823|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|DMGenerateLogInterceptor - InternalTransactionID:SDP-DM-31297182, ExternalTransactionID:1283918818906926, TransactionStatus:Sent For Delivery, INStatus:Success
08-Sep-2010 09:47:52,823|9122796686|internal_app|1283918818906926|SendSms|ChargeAmount|Response Parameters:[Result:Success]
08-Sep-2010 09:47:52,970|BJ#8748783|phtunes_app||SendSms|Request Params:[Address:tel:BJ#8748783,Message:Joke:  Jack: What are you going to be when you have finished studying and passing all your exams. Buck: Probably an old age pensioner.                          ,SenderName:52211,ChargingInformation:[Code:code0,Currency:INR,Description:CP is Calling TO This API,Amount:0],SimpleReference:[],X-Params:[[Key:transactionId,Value:1283918819184906],[Key:origContentOwnerId,Value:10225],[Key:channelType,Value:SMS],[Key:cpID,Value:Phoneytunes],[Key:serviceId,Value:PT_JOKES],[Key:PPL_FLAG,Value:FALSE],[Key:contentType,Value:Text],[Key:contentName,Value:PT_JOKES],[Key:senderMSISDN,Value:52211],[Key:isRated,Value:No],[Key:contentId,Value:1096]]]
08-Sep-2010 09:47:52,970|BJ#8748783|phtunes_app||SendSms|ValidateMSISDNInterceptor - sid to be validated BJ#8748783
In the above logs i want to search a list of strings which are available in an array as follows:
shortcodes=( "56882" "58585" "58888" "57575" "57677" );
Now I want to search the string contains "ShortCode=tel:"$shortcode from the last 15 minutes logs and calculate their count accordingly.For example
56882 : 12
58585 : 15
58888 : 18
57575 : 12
57677 : 98
The above output includes sum of count matching the string from both files on different servers.

The code what i have written is as follows and giving error while accessing file on other server:

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-90000
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "52211" );
for shortcode in ${shortcodes[@]}
    sh_count=`awk '$0>=from && $0<=to && $0~"ShortCode=tel:" s' from="$from" to="$to" s="$shortcode" App_OP.log | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh root@ -n "awk '$0>=from && $0<=to && $0~"ShortCode=tel:" s' from="$from" to="$to" s="$shortcode" App_OP.log | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
Please guide me for the above and if possible provide me more optimized way of doing above.
Thanks in advance
Old 09-08-2010, 06:59 AM   #2
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,607

Rep: Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935Reputation: 2935
Well I see a number of issues with this code. Some to make it easier and some that are just wrong.

Easier use of from calculation:
from=$(date -d '15 min ago' +"%d-%b-%Y %T")
Your awk statements are flawed:
sh_count=`awk '$0>=from && $0<=to && $0~"ShortCode=tel:" s' from="$from" to="$to" s="$shortcode" App_OP.log | wc -l`
$0 is the entire line from the log file so it is going to generate inaccurate results when testing <= or >= to a time as only the start of the line contains
this information. As the information is trivial, you may wish to set the field separator as a comma and then use $1 for the from / to tests.
sh_count=`awk -F, '$1>=from && $1<=to && $0~"ShortCode=tel:" s' from="$from" to="$to" s="$shortcode" App_OP.log | wc -l`
Also, why not just have the awk count your items instead of using wc which is a call to another external program, hence slowing the script down.

Your counter increases do not require the extra calls to expr:
And yes, you do not require the $ symbol

Last edited by grail; 09-08-2010 at 07:00 AM.



Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Basic issue: Accessing files between computers baldurpet Linux - Server 5 01-31-2009 08:16 AM
SELinux issue. (httpd & samba accessing same files and directories) vinten Linux - Security 2 01-08-2009 10:40 AM
tcpdump - issue faced with dumping UDP packets tanmay.anjaria Linux - Newbie 2 02-09-2008 10:38 PM
Accessing servers from the internet... AudioMechanic Linux - Networking 1 03-17-2005 09:05 PM
Issue copying data files from CD fryak Fedora 3 05-28-2004 11:49 AM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:57 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration