Hi ,

I am trying to combine two awk commands to retrieve dateTime,userID,email,ip etc in the below format

Expected Output

2018-08-15 21:15:03 clientname programid participantId participantFirstName

The individual awk command works fine but when combined together getting a blank result. How can the script be improved to get the desired results.Please assist.



<Code>

ssh -i /home/username/.ssh/id_rsa username@xx.xxx.xx.xx 'zgrep -i -h -B8 "Decision from API: DECLINED" /usr/local/apache-tomcat/logs/archive/logFilename.log.2018-08-15.gz' \ | grep "API fraud detection is enabled" | awk -F'[][]' '{print $2}'|awk -F '[[:blank:],]' ' {print $21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31}'|uniq

</Code>


Log file

[2018-08-15 21:15:03,629] hostname 1534382111429 ClientName:FF_CC:6980771296 http-nio-8080-exec-23 INFO 6980771296 ClientName FF_CC: mark a pending order.{30001MP032LL/A:Electronics, Electronics Items:1} (OrderCommitStatusService)
[2018-08-15 21:15:03,630] xx.xx.xx.net 1534382111429 ClientName:FF_CC:6980771296 http-nio-8080-exec-23 INFO API fraud detection is enabled. Sending the API request: com.client.API.model.APIRequest@6a5f3f67[total=30411,ccAmount=0,varId=ClientName,programId=FF_CC,participantId=6980771296,participantName=Mer yl Stout,email=username@xx.COM,agentId=<null>,ipAddress=xx.xx.xx.xx,shippingPhoneNumber=(xx) xxx-xxx,address=com.API.ris.util.Address@3cd66be4,APISession=com.client.API.model.APISession@12c6abe5,cu rrency=USD,merchantAcknowledgment=Y,payment=com.API.ris.util.payment.NoPayment@427f6ea1,cart=[Product Type: Electronics AA
Item Name: Electronics, Electronics Items
Description: Electronics, Electronics Items
Quantity: 1
Price: 27900

]] (APIService)
[2018-08-15 21:15:04,489] app1.xx.xx.net 1534382111429 Delta:FF_CC:6980771296 http-nio-8080-exec-23 INFO Decision from API: DECLINED (APIService)

Something like :
Remove grep and one awk only.

Hi lougavulin,

Thanks for the quick response.I forget to mention that i have initially written the below condition higlighted in the script, which resulted in getting the below results.

VarId programId participantId participantFirstName participantLastName email ipAddress UScode Shipping Number

Initial Code

<Code>
ssh -i /home/tomcat/.ssh/id_rsa tomcat@${webAppservers[i]} 'grep -i -h -B8 "Decision from API: DECLINED" /usr/local/apache-tomcat/logs/logfile.log.'$dt'.gz' \ | grep "API fraud detection is enabled" | awk -F '[[:blank:],]' ' {print $21,$22,$23,$24,$25,$26,$28,$29,$30}'|uniq >>poc.txt
sed -ri 's/varId=|programId=|participantId=|participantName=|email=|ipAddress=|shippingPhoneNumber=//g' poc.txt
</Code>


Incorporated the recommendation

<Code>
ssh -i /home/tomcat/.ssh/id_rsa tomcat@${webAppservers[i]} 'grep -i -h -B8 "Decision from API: DECLINED" /usr/local/apache-tomcat/logs/archive/logfile.log.'$dt'.gz' | awk -F'[][,]' '/API fraud detection is enabled/ {split($4, aFOUR, " "); print $2, $3, aFOUR[3], $8, $9, $10 ; }' | uniq >> poc.txt
sed -ri 's/varId=|programId=|participantId=|participantName=|email=|ipAddress=|shippingPhoneNumber=//g' poc.txt
</Code>

And got the below results as below
2018-08-15 21:15:03 630 Delta:FF_CC:6980771296 FF_CC 6980771296 Meryl Stout

How can i get the date and time in the result as below. Sorry for the inconvenience.

2018-08-15 21:15:03 Delta FF_CC 6980771296 Meryl Stout xx@GMAIL.COM xx.xx.xx.xx (xx) xx-xx

Please do not use <Code> but [code] instead.

do not use two greps and one or two awks and a sed together, one single awk and a [z]cat should be enough:
and the awk script will run on the local host (something like this):

Well, maybe there is a nicer way to do, but here is a working stuf :
zcat instead of (z)grep, remove last sed and one awk.

1 members found this post helpful.

Hi lougavulin,

This worked, but i am unable to understand the logic inside the code.Could you please explain for the reference.Again much thanks for the work around.
{ MAIN=$2" %DFAD% "$12" "$14" "$16" "$18" "$22" "$24 ; } /Decision from API: DECLINED/ { split($4, aFOUR, "[ :]"); sub("%DFAD%", aFOUR[4], MAIN); print MAIN;}' | uniq >poc.txt

Your log file put information you need in 1 line on 2 lines, which are not close to each other.

There is 2 parts, each start with :
Awk applies 'foobar' only for lines with in 'blahblah'.

So with the first part :
This part deal with the first line, the main one, with most information.
It builds one string (MAIN) with all possibles information, formatted as you want where you want.
As the missing information is in the middle, temporally, it puts the missing one as '%DFAD%'.

So when it leaves this part, you have a string like that :
2018-08-15 21:15:03 %DFAD% FF_CC 6980771296 Meryl Stout xx@GMAIL.COM xx.xx.xx.xx (xx) xx-xx
But it do not print it for you to see.

The second part, starting :
The first thing is to get the missing information.
The field separators did not change, so we still use '][,='. The missing information is in $4 field. To get it, it split $4 into an array using '<space>' and ':' as separators. So what you want is in the 4th array's cell.

It replace '%DFAD%' by the 4th array's cell into the main string.

And as the lines are in order and it has all information you want, it prints the result :
2018-08-15 21:15:03 Delta FF_CC 6980771296 Meryl Stout xx@GMAIL.COM xx.xx.xx.xx (xx) xx-xx

I tried to be clear...

1 members found this post helpful.

Once more, please put your code inside [CODE] (via the sharp sign icon in the editor). You can edit your previous posts with the "Edit" button at the bottom right of your own each post.