LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
Reload this Page awk help - matching
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-17-2007, 01:37 AM   #1
uttam_h
LQ Newbie
 
Registered: Sep 2002
Location: mangalore,india
Distribution: redhat
Posts: 28

Rep: Reputation: 15
awk help - matching

(Please read my previous post "awk help")

Hi all,
I have a log file with 2 types of blocks
1) clientRequest
2) clientRequestResponse


####<Aug 30, 2007 10:56:30 AM IST><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<clientRequest xmlns:m="http://www.openuri.org/">
..................
.................
.................
</clientRequest>

####<Aug 29, 2007 10:56:30 AM IST><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<clientRequestResponse xmlns:m="http://www.openuri.org/">
...............
..................
..................
</clientRequestResponse>


Now using sed/awk i want to extract
1)all the clientRequest block (also m:clientRequestResponse,ns:clientRequestResponse etc)

2)all the clientRequestResponse block (also m:clientRequestResponse,ns:clientRequestResponse etc)

3)all the clientRequest block for a given date eg Aug 30, 2007 (also m:clientRequest,ns:clientRequest etc)

4)all the clientRequestResponse block for a given date eg Aug 30, 2007
(also m:clientRequest,ns:clientRequest etc)


Thanks and Regards,
uttam hoode

Code:
####<Aug 30, 2007 10:56:30 AM IST> <Info> <ALSB Logging> <saturn> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1188451590854> <000000> < [Name_ID_Mapping, Name_ID_Mapping_request, Mapping ID_Name, REQUEST] +++++++++NProxy Request++++++++++: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <m:clientRequest xmlns:m="http://www.openuri.org/">
    <env:EaiEnvelope xmlns:env="http://eai.aaan.in/Envelope">
      <env:Domain>AuctionPortal</env:Domain>
      <env:Service>ece</env:Service>
      <env:Sender>abl_ap</env:Sender>
      <env:MessageId>aaaa</env:MessageId>
      <env:Language>en</env:Language>
      <env:UserId>abl_ap</env:UserId>
      <env:CorrelationId>A001</env:CorrelationId>
      <env:Payload>
        <ec:eceData xmlns:ec="http://eai.aaa.in/ece">
          <ec:Request>
            <ec:Operation_Name>authenticateCustomerForAuctionPortal</ec:Operation_Name>
            <ec:CustDetails_InputData>
              <ec:MSISDN>989880000065</ec:MSISDN>
              <ec:language>en</ec:language>
              <ec:newPassword>password</ec:newPassword>
            </ec:CustDetails_InputData>
          </ec:Request>
        </ec:eceData>
      </env:Payload>
    </env:EaiEnvelope>
  </m:clientRequest>
</soapenv:Body>> 

####<Aug 30, 2007 10:56:31 AM IST> <Info> <ALSB Logging> <saturn> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1188451591466> <000000> < [Name_ID_Mapping, Name_ID_Mapping_response, Response Stage, RESPONSE] +++++++NPResponse+++++++++: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <m:clientRequestResponse xmlns:m="http://www.openuri.org/">
    <env:EaiEnvelope xmlns:env="http://eai.aaan.in/Envelope">
      <env:Domain>AuctionPortal</env:Domain>
      <env:Service>ece</env:Service>
      <env:Sender>abl_ap</env:Sender>
      <env:MessageId>aaaI_AP</env:MessageId>
      <env:Language>en</env:Language>
      <env:UserId>abl_ap</env:UserId>
      <env:CorrelationId>A001</env:CorrelationId>
      <env:Payload>
        <ns1:eceData xmlns:ns1="http://eai.aaa.in/ece" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
          <ns1:Response>
            <ns1:Result_OutputData>
              <ns1:resultCode>0</ns1:resultCode>
              <ns1:resultMessage>Operation Completed Successfully.</ns1:resultMessage>
              <ns1:reference_ID>1</ns1:reference_ID>
            </ns1:Result_OutputData>
            <ns1:CustDetails_OutputData>
              <ns1:firstName>N</ns1:firstName>
              <ns1:lastName>N</ns1:lastName>
              <ns1:Customer_Type>N</ns1:Customer_Type>
              <ns1:status>AC</ns1:status>
              <ns1:isLocked>false</ns1:isLocked>
              <ns1:isBlocked>false</ns1:isBlocked>
              <ns1:isFirstLogin>true</ns1:isFirstLogin>
              <ns1:hintQuestion>test</ns1:hintQuestion>
              <ns1:hintAnswer>test</ns1:hintAnswer>
              <ns1:lastLoginDateTime>2007-08-30T05:14:23.000Z</ns1:lastLoginDateTime>
              <ns1:BSSUBSCODE>99999</ns1:BSSUBSCODE>
              <ns1:ACCOUNT_NO>99999</ns1:ACCOUNT_NO>
              <ns1:isAuction>Y</ns1:isAuction>
            </ns1:CustDetails_OutputData>
          </ns1:Response>
        </ns1:eceData>
      </env:Payload>
    </env:EaiEnvelope>
  </m:clientRequestResponse>
</soapenv:Body>> 

####<Aug 30, 2007 10:57:27 AM IST> <Info> <ALSB Logging> <saturn> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1188451647464> <000000> < [Name_ID_Mapping, Name_ID_Mapping_request, Mapping ID_Name, REQUEST] +++++++++NProxy Request++++++++++: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <m:clientRequest xmlns:m="http://www.openuri.org/">
    <env:EaiEnvelope xmlns:env="http://eai.aaan.in/Envelope">
      <env:Domain>MobileLoadedPortal</env:Domain>
      <env:Service>ece</env:Service>
      <env:Sender>abl_mp</env:Sender>
      <env:MessageId>aaaI_MLP</env:MessageId>
      <env:Language>en</env:Language>
      <env:UserId>abl_mp</env:UserId>
      <env:CorrelationId>A001</env:CorrelationId>
      <env:Payload>
        <ec:eceData xmlns:ec="http://eai.aaa.in/ece">
          <ec:Request>
            <ec:Operation_Name>getCustomerInfo</ec:Operation_Name>
            <ec:CustDetails_InputData>
              <ec:MSISDN>989880000065</ec:MSISDN>
              <ec:language>en</ec:language>
            </ec:CustDetails_InputData>
          </ec:Request>
        </ec:eceData>
      </env:Payload>
    </env:EaiEnvelope>
  </m:clientRequest>
</soapenv:Body>>
Last edited by uttam_h; 10-17-2007 at 01:40 AM.
 
Old 10-17-2007, 04:33 AM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
Sorry, but what do you mean when you say you "want to extract
a block"? You're trying to split the file into chunks?



Can you provide a sample of what you expect to see after
that "extraction"?


Cheers,
Tink
 
Old 10-17-2007, 04:54 AM   #3
uttam_h
LQ Newbie
 
Registered: Sep 2002
Location: mangalore,india
Distribution: redhat
Posts: 28

Original Poster
Rep: Reputation: 15
there are 2 types of block in the above log

1) clientRequest

Code:
####<Aug 30, 2007 10:56:30 AM IST> <Info> <ALSB Logging> <saturn> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1188451590854> <000000> < [Name_ID_Mapping, Name_ID_Mapping_request, Mapping ID_Name, REQUEST] +++++++++NProxy Request++++++++++: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <m:clientRequest xmlns:m="http://www.openuri.org/">
    <env:EaiEnvelope xmlns:env="http://eai.aaan.in/Envelope">
      <env:Domain>AuctionPortal</env:Domain>
      <env:Service>ece</env:Service>
      <env:Sender>abl_ap</env:Sender>
      <env:MessageId>aaaa</env:MessageId>
      <env:Language>en</env:Language>
      <env:UserId>abl_ap</env:UserId>
      <env:CorrelationId>A001</env:CorrelationId>
      <env:Payload>
        <ec:eceData xmlns:ec="http://eai.aaa.in/ece">
          <ec:Request>
            <ec:Operation_Name>authenticateCustomerForAuctionPortal</ec:Operation_Name>
            <ec:CustDetails_InputData>
              <ec:MSISDN>989880000065</ec:MSISDN>
              <ec:language>en</ec:language>
              <ec:newPassword>password</ec:newPassword>
            </ec:CustDetails_InputData>
          </ec:Request>
        </ec:eceData>
      </env:Payload>
    </env:EaiEnvelope>
  </m:clientRequest>
</soapenv:Body>>
2) clientRequestResponse

Code:
####<Aug 30, 2007 10:56:31 AM IST> <Info> <ALSB Logging> <saturn> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1188451591466> <000000> < [Name_ID_Mapping, Name_ID_Mapping_response, Response Stage, RESPONSE] +++++++NPResponse+++++++++: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <m:clientRequestResponse xmlns:m="http://www.openuri.org/">
    <env:EaiEnvelope xmlns:env="http://eai.aaan.in/Envelope">
      <env:Domain>AuctionPortal</env:Domain>
      <env:Service>ece</env:Service>
      <env:Sender>abl_ap</env:Sender>
      <env:MessageId>aaaI_AP</env:MessageId>
      <env:Language>en</env:Language>
      <env:UserId>abl_ap</env:UserId>
      <env:CorrelationId>A001</env:CorrelationId>
      <env:Payload>
        <ns1:eceData xmlns:ns1="http://eai.aaa.in/ece" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
          <ns1:Response>
            <ns1:Result_OutputData>
              <ns1:resultCode>0</ns1:resultCode>
              <ns1:resultMessage>Operation Completed Successfully.</ns1:resultMessage>
              <ns1:reference_ID>1</ns1:reference_ID>
            </ns1:Result_OutputData>
            <ns1:CustDetails_OutputData>
              <ns1:firstName>N</ns1:firstName>
              <ns1:lastName>N</ns1:lastName>
              <ns1:Customer_Type>N</ns1:Customer_Type>
              <ns1:status>AC</ns1:status>
              <ns1:isLocked>false</ns1:isLocked>
              <ns1:isBlocked>false</ns1:isBlocked>
              <ns1:isFirstLogin>true</ns1:isFirstLogin>
              <ns1:hintQuestion>test</ns1:hintQuestion>
              <ns1:hintAnswer>test</ns1:hintAnswer>
              <ns1:lastLoginDateTime>2007-08-30T05:14:23.000Z</ns1:lastLoginDateTime>
              <ns1:BSSUBSCODE>99999</ns1:BSSUBSCODE>
              <ns1:ACCOUNT_NO>99999</ns1:ACCOUNT_NO>
              <ns1:isAuction>Y</ns1:isAuction>
            </ns1:CustDetails_OutputData>
          </ns1:Response>
        </ns1:eceData>
      </env:Payload>
    </env:EaiEnvelope>
  </m:clientRequestResponse>
</soapenv:Body>>

log has many such entries....using awk i want to extract clientRequestResponse or clientRequest. also based on date

Regards,
uttam hoode
 
Old 10-17-2007, 05:03 AM   #4
amitsurana
LQ Newbie
 
Registered: Oct 2007
Location: India
Distribution: ubuntu, redhat
Posts: 5
Blog Entries: 1

Rep: Reputation: 0
You can consider "<" as your field separator cos every new Tag starts with "<" .

Hence :

awk -F "<" ClientRequest -> this will give all the Log entries...

I hope u can get my idea....
 
Old 10-17-2007, 05:30 AM   #5
yongitz
Member
 
Registered: Nov 2005
Location: Davao City, Philippines
Distribution: RHEL, CentOS, Ubuntu, Mint
Posts: 139

Rep: Reputation: 20
Hi! you could use this post as your reference. This would give you an idea to get started...

http://www.linuxquestions.org/questi...e-line-592417/
 
Old 10-17-2007, 05:44 AM   #6
radoulov
Member
 
Registered: Apr 2007
Location: Milano, Italia/Варна, България
Distribution: Ubuntu, Open SUSE
Posts: 212

Rep: Reputation: 38
_If I understand correctly_ :

1. To extract the different blocks:

clientRequest

Code:
awk '/^###/{x=$0}
$0~v{print x"\n"$0;f=1;next}
f&&/<\/soapenv/{print;f=0}f
' v="<m:clientRequest " filename

clientRequestResponse

Code:
awk '/^###/{x=$0}
$0~v{print x"\n"$0;f=1;next}
f&&/<\/soapenv/{print;f=0}f
' v="<m:clientRequestR" filename
clientRequest for a given date:

Code:
awk '/^###/&&$0~dt{x=$0;f1=1}
f1&&$0~v{print x"\n"$0;f=1;next}
f&&/<\/soapenv/{print;f=f1=0}f
' dt="Aug 23, 2007" v="<m:clientRequest " filename
clientRequestResponse for a given date:

Code:
awk '/^###/&&$0~dt{x=$0;f1=1}
f1&&$0~v{print x"\n"$0;f=1;next}
f&&/<\/soapenv/{print;f=f1=0}f
' dt="Aug 30, 2007" v="<m:clientRequestR" filename
 
Old 10-17-2007, 01:20 PM   #7
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
Quote:
Originally Posted by uttam_h View Post
there are 2 types of block in the above log

1) clientRequest

2) clientRequestResponse

log has many such entries....using awk i want to extract clientRequestResponse or clientRequest. also based on date
Yes, I had seen those...

You *still* didn't say what exactly you mean by extract,
but I'll assume that you're trying to split out the whole
chunks w/o any reformatting.


If the structure of the log is consistently like the
tiny snippet you provided the easiest way to deal
with it would be like so:


Code:
awk 'BEGIN{RS="\n\n"; FS="\n"; ORS=RS; OFS=FS} /<Aug 30, 2007/ && /clientRequestResponse /' log


Cheers,
Tink
Last edited by Tinkster; 10-17-2007 at 01:22 PM. Reason: typo
 
Old 10-17-2007, 03:20 PM   #8
radoulov
Member
 
Registered: Apr 2007
Location: Milano, Italia/Варна, България
Distribution: Ubuntu, Open SUSE
Posts: 212

Rep: Reputation: 38
Quote:
Originally Posted by Tinkster View Post
[...]
If the structure of the log is consistently like the
tiny snippet you provided the easiest way to deal
with it would be like so:


Code:
awk 'BEGIN{RS="\n\n"; FS="\n"; ORS=RS; OFS=FS} /<Aug 30, 2007/ && /clientRequestResponse /' log
Or just:

Code:
awk '/<Aug 30, 2007/&&/clientRequestResponse /' RS="\n\n" file
As already stated there are a few particularities here:

1. The above solution works only with GNU Awk (and may be tawk,
cannot check right now), but this is a Linux forum, so that's ok
2. Must pay attention for possible "record too long" errors
(not with the given format, of course).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Find/grep command to find matching files, print filename, then print matching content stefanlasiewski Programming 9 06-30-2016 06:30 PM
AWK/SED Multiple pattern matching over multiple lines issue GigerMalmensteen Programming 15 12-03-2006 06:08 PM
sed/awk for matching & substituition laikos Programming 3 11-22-2006 12:44 PM
awk/gawk/sed - read lines from file1, comment out or delete matching lines in file2 rascal84 Linux - General 1 05-24-2006 10:19 AM
awk: /matching/ variables passed with -v aunquarra Linux - General 2 02-17-2005 07:47 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:00 PM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration