[SOLVED] how to merge three awk to one?

Aeolustw · 10-09-2016, 07:02 PM

I use awk in bash shell to analyze the syslog. And catch ip match what i want, somthing like

Code:

    #!/bin/bash
    awk -F'[#]|client ' '/query.*denied/{a[$2];b[$2]++}END{for(i in a)if(b[i]>0)printf "%-15s %-27s %-2s\n",i, ",query_denied", b[i]}' /var/log/syslog.1 > output
    awk -F'[()]|smtp:|submission:' '/max connection count/{a[$3];b[$3]++}END{for(i in a)if(b[i]>0)printf "%-15s %-27s %-2s\n",i, ",max_connection_count", b[i]}' /var/log/syslog.1 >> output
    awk -F'[][]' '/SSL_accept error from unknown/{a[$4];b[$4]++}END{for(i in a)if(b[i]>0)printf "%-15s %-27s %-2s\n",i, ",SSL_accept_error", b[i]}' /var/log/syslog.1 >> output

Is it possible to merge these three awk to one awk,like?

Code:

    #!/bin/bash
    awk -F'[#][()]|client|smtp:|submission:' '....' > output

/var/log/syslog.1

Code:

    Oct  7 02:21:48 ipb named[2677]: client 38.229.33.47#59569: query (cache) 'a998207098p59569i39337.d2016100618000222958.t12135.dnsresearch.cymru.com/A/IN' denied
    Oct  7 02:39:12 ipb named[2677]: client 183.56.172.145#20000: query (cache) '2054061883.www.baidu.com/A/IN' denied
    Oct  7 04:31:44 ipb named[2677]: client 141.212.122.111#38457: query (cache) 'c.afekv.com/A/IN' denied
    Oct  7 05:34:21 ipb named[2677]: client 95.215.60.214#43977: query (cache) 'm24.pl/ANY/IN' denied
    Oct  7 06:39:09 ipb named[2677]: client 185.94.111.1#46130: query (cache) 'com/ANY/IN' denied
    Oct  7 08:22:08 ipb named[2677]: client 209.126.136.2#52517: query (cache) 'a.gtld-servers.net/A/IN' denied
    Oct  7 09:00:09 ipb named[2677]: client 185.141.24.209#42825: query (cache) 'leth.cc/ANY/IN' denied
    Oct  7 09:28:25 ipb named[2677]: client 124.232.142.220#38773: query (cache) 'www.google.com/A/IN' denied
    Oct  7 12:31:08 ipb named[2677]: client 124.232.142.220#38332: query (cache) 'www.google.it/A/IN' denied
    Oct  7 01:36:57 ipb postfix/anvil[15006]: statistics: max connection count 1 for (smtp:223.74.42.35) at Oct  7 01:33:36
    Oct  7 03:14:45 ipb postfix/anvil[13320]: statistics: max connection count 1 for (submission:169.56.71.47) at Oct  7 03:11:24
    Oct  7 04:16:04 ipb postfix/anvil[7596]: statistics: max connection count 1 for (smtp:223.74.42.155) at Oct  7 04:12:43
    Oct  7 09:03:20 ipb postfix/anvil[357]: statistics: max connection count 1 for (smtp:62.219.225.141) at Oct  7 09:00:00
    Oct  7 11:47:26 ipb postfix/anvil[28328]: statistics: max connection count 1 for (smtp:81.240.248.53) at Oct  7 11:44:03
    Oct  7 13:54:54 ipb postfix/anvil[1113]: statistics: max connection count 1 for (smtp:210.211.102.38) at Oct  7 13:51:33
    Oct  7 22:28:26 ipb postfix/anvil[31118]: statistics: max connection count 1 for (smtp:80.82.64.102) at Oct  7 22:25:00
    Oct  7 03:11:25 ipb postfix/submission/smtpd[13318]: SSL_accept error from unknown[169.56.71.47]: lost connection

output

Code:

    141.212.122.111 ,query_denied               1 
    38.229.33.47    ,query_denied               1 
    124.232.142.220 ,query_denied               2 
    183.56.172.145  ,query_denied               1 
    209.126.136.2   ,query_denied               1 
    95.215.60.214   ,query_denied               1 
    185.94.111.1    ,query_denied               1 
    185.141.24.209  ,query_denied               1 
    80.82.64.102    ,max_connection_count       1 
    169.56.71.47    ,max_connection_count       1 
    62.219.225.141  ,max_connection_count       1 
    223.74.42.35    ,max_connection_count       1 
    81.240.248.53   ,max_connection_count       1 
    210.211.102.38  ,max_connection_count       1 
    223.74.42.155   ,max_connection_count       1 
    169.56.71.47    ,SSL_accept_error           1

I have no idea for this, After search google, still no help. any hint?
Thanks.

Best Regard.

Turbocapitalist · 10-09-2016, 09:27 PM

Yes, it is possible, but I think the approach of changing the Field Separator is not helpful. I would leave the FS as whitespace and work on finding the right patterns followed by a little manipulation with gsub or something. Then you'd have to collect your cumulative stats in a different array for each of the three categories.

Code:

#!/usr/bin/awk -f

/situation1/ { something1 }
/situation2/ { something2 }
/situation3/ { something3 }

END {
        for(i in a) { summary1 }
        for(i in b) { summary2 }
        for(i in c) { summary3 }
}

Also, which version of "awk" are you using for this?

syg00 · 10-09-2016, 10:18 PM

Quote:

Originally Posted by Aeolustw

Is it possible to merge these three awk to one awk,like?

Code:

    #!/bin/bash
    awk -F'[#][()]|client|smtp:|submission:' '....' > output

How would you know what fields ($2, $3, ...) represent. Much better option offered above.

pan64 · 10-10-2016, 01:46 AM

Personally I would go with perl, that would be quite simple, but you can do it in awk too.
In awk I would use one, single (common) FS for all the cases you have - containing []()#: and whitespace and you will only need to handle the required columns (as it was described in post #2)

goumba · 10-10-2016, 03:13 AM

Why the need for a one liner? Not for nothin', but your three lines are hard enough to read as is, and I'd hate to have to debug or modify it.

awk lets you run scripts from a file. If you are concerned about cluttering your bash script, while not follow along Turbocapitalist's lead, and put that in a separate awk script?

myscript.sh

Code:

#!/bin/bash
... stuff ...
awk -f myawk.awk
... rest of script ...

get_ip.awk

Code:

#!/usr/bin/awk -f

/query.*denied/{...processing...}
/(smtp|submission):/{...processing...}
/SSL_accept error/{...processing...}

END { print $discovered_ip }

The hash bang lets you run the script as simply ./get_ip.awk as well.

danielbmartin · 10-11-2016, 07:39 AM

This is a brute force "mashup" of the OP's code.

With this InFile ...

Code:

    Oct  7 02:21:48 ipb named[2677]: client 38.229.33.47#59569: query (cache) 'a998207098p59569i39337.d2016100618000222958.t12135.dnsresearch.cymru.com/A/IN' denied
    Oct  7 02:39:12 ipb named[2677]: client 183.56.172.145#20000: query (cache) '2054061883.www.baidu.com/A/IN' denied
    Oct  7 04:31:44 ipb named[2677]: client 141.212.122.111#38457: query (cache) 'c.afekv.com/A/IN' denied
    Oct  7 05:34:21 ipb named[2677]: client 95.215.60.214#43977: query (cache) 'm24.pl/ANY/IN' denied
    Oct  7 06:39:09 ipb named[2677]: client 185.94.111.1#46130: query (cache) 'com/ANY/IN' denied
    Oct  7 08:22:08 ipb named[2677]: client 209.126.136.2#52517: query (cache) 'a.gtld-servers.net/A/IN' denied
    Oct  7 09:00:09 ipb named[2677]: client 185.141.24.209#42825: query (cache) 'leth.cc/ANY/IN' denied
    Oct  7 09:28:25 ipb named[2677]: client 124.232.142.220#38773: query (cache) 'www.google.com/A/IN' denied
    Oct  7 12:31:08 ipb named[2677]: client 124.232.142.220#38332: query (cache) 'www.google.it/A/IN' denied
    Oct  7 01:36:57 ipb postfix/anvil[15006]: statistics: max connection count 1 for (smtp:223.74.42.35) at Oct  7 01:33:36
    Oct  7 03:14:45 ipb postfix/anvil[13320]: statistics: max connection count 1 for (submission:169.56.71.47) at Oct  7 03:11:24
    Oct  7 04:16:04 ipb postfix/anvil[7596]: statistics: max connection count 1 for (smtp:223.74.42.155) at Oct  7 04:12:43
    Oct  7 09:03:20 ipb postfix/anvil[357]: statistics: max connection count 1 for (smtp:62.219.225.141) at Oct  7 09:00:00
    Oct  7 11:47:26 ipb postfix/anvil[28328]: statistics: max connection count 1 for (smtp:81.240.248.53) at Oct  7 11:44:03
    Oct  7 13:54:54 ipb postfix/anvil[1113]: statistics: max connection count 1 for (smtp:210.211.102.38) at Oct  7 13:51:33
    Oct  7 22:28:26 ipb postfix/anvil[31118]: statistics: max connection count 1 for (smtp:80.82.64.102) at Oct  7 22:25:00
    Oct  7 03:11:25 ipb postfix/submission/smtpd[13318]: SSL_accept error from unknown[169.56.71.47]: lost connection#!/bin/

... this awk ...

Code:

awk -F'client |[][)(#]|smtp:|submission:|SSL_accept error from unknown' \
    '{/denied/        ?a[$4]++:0;
      /max connection/?b[$5]++:0;
      /accept/        ?c[$5]++:0;} 
 END{for(i in a) printf "%-15s%-27s%-2s\n",i,",query_denied",a[i]
     for(i in b) printf "%-15s%-27s%-2s\n",i,",max_connection_count",b[i]
     for(i in c) printf "%-15s%-27s%-2s\n",i,",SSL_accept_error",c[i]}' \
 $InFile >$OutFile

... produced this OutFile ...

Code:

141.212.122.111 ,query_denied               1 
38.229.33.47    ,query_denied               1 
124.232.142.220 ,query_denied               2 
183.56.172.145  ,query_denied               1 
209.126.136.2   ,query_denied               1 
95.215.60.214   ,query_denied               1 
185.94.111.1    ,query_denied               1 
185.141.24.209  ,query_denied               1 
80.82.64.102    ,max_connection_count       1 
169.56.71.47    ,max_connection_count       1 
62.219.225.141  ,max_connection_count       1 
223.74.42.35    ,max_connection_count       1 
81.240.248.53   ,max_connection_count       1 
210.211.102.38  ,max_connection_count       1 
223.74.42.155   ,max_connection_count       1 
169.56.71.47    ,SSL_accept_error           1

Daniel B. Martin

Aeolustw · 10-11-2016, 07:02 PM

@Turbocapitalist:
Thanks for the useful basic type codes.

My awk version:

Code:

awk -Wversion 2>/dev/null || awk --version

GNU Awk 4.0.1

@syg00
When I check the syslog, then i will know what match field it is, like

Code:

awk -F'[#]|client ' '/query.*denied/{a[$2];b[$2].....

@goumba
Thanks for the hint and useful codes.

@danielbmartin
I am so glad to see what it is clean and short.
Thanks for help.

Finally, i also have done it:

Code:

awk -F'[][()#=/,]|client |smtp:|submission:|unknown' \
'/query.*denied/{a[$4];b[$4]++;next}
/max connection count/{c[$6];d[$6]++;next}
(/SSL_accept error from unknown/ && /\/submission\//){e[$7];f[$7]++;next}
(/SSL_accept error from unknown/ && !/\/submission\//){g[$6];h[$6]++}
END {
for(i in a){if(b[i]>0){printf "%-15s %-27s %-2s\n",i, ",query_denied", b[i]}}
for(j in c){if(d[j]>0){printf "%-15s %-27s %-2s\n",j, ",max_connection_count", d[j]}}
for(k in e){if(f[k]>0){printf "%-15s %-27s %-2s\n",k, ",SSL_accept_error", f[k]}}
for(l in g){if(h[l]>0){printf "%-15s %-27s %-2s\n",l, ",SSL_accept_error", h[l]}}}' \
$InFile >$OutFile

There are two different syslog type,like

Quote:

Sep 11 13:56:12 ipb postfix/submission/smtpd[23054]: disconnect from unknown[94.102.49.193]
Sep 11 13:59:32 ipb postfix/anvil[23056]: statistics: max connection rate 1/60s for (submission:94.102.49.193) at Sep 11 13:56:08

Thanks all guys for your help.

grail · 10-12-2016, 09:04 AM

Try putting it into a file and running as a script so you can comment it and understand what all the tasks are you performed in a months time

Here is a slightly longer version in script format so you can see how laying it out is easier to follow (I'll leave the comments to you

)

Code:

#!/usr/bin/awk -f

{
  regex = "[^0-9.]"

  switch ($0){
    case /query/:
      if($NF == "denied"){
        field = $7
        name  = "query_denied"
        regex = "#.*"
      }
      break
    case /max connection/:
      field = $(NF-4)
      name  = "max_connection_count"
      break
    case /SSL_accept/:
      field = $(NF-2)
      name  = "SSL_accept_error"
  }
  ip = gensub(regex,"","g",field)

  output[name][ip]++
}

END{
  for(i in output)
    for(j in output[i])
      print j","i,output[i][j]
}

I find well named variables can also help eliminate the need for extensive comments

Aeolustw · 10-13-2016, 07:07 AM

Thanks.

I add the default case for if no match, and edit print to printf.

Code:

#!/usr/bin/awk -f

{
  regex = "[^0-9.]"

  switch ($0){
    case /query/:
      if($NF == "denied"){
        field = $7
        name  = "query_denied"
        regex = "#.*"
      }
      break
    case /max connection/:
      field = $(NF-4)
      name  = "max_connection_count"
      break
    case /SSL_accept/:
      field = $(NF-2)
      name  = "SSL_accept_error"
      break
    default:
      field = $(NF-2)
      name  = "No_match"
      break
  }
  ip = gensub(regex,"","g",field)

  output[name][ip]++
}

END{
 for(i in output)
    for(j in output[i])
    	printf "%-15s %-27s %-2s\n",j,","i,output[i][j]
}

Aeolustw · 10-17-2016, 07:34 PM

Quote:

Originally Posted by grail

Try putting it into a file and running as a script so you can comment it and understand what all the tasks are you performed in a months time

Here is a slightly longer version in script format so you can see how laying it out is easier to follow (I'll leave the comments to you

)

Code:

#!/usr/bin/awk -f

{
  regex = "[^0-9.]"

  switch ($0){
    case /query/:
      if($NF == "denied"){
        field = $7
        name  = "query_denied"
        regex = "#.*"
      }
      break
    case /max connection/:
      field = $(NF-4)
      name  = "max_connection_count"
      break
    case /SSL_accept/:
      field = $(NF-2)
      name  = "SSL_accept_error"
  }
  ip = gensub(regex,"","g",field)

  output[name][ip]++
}

END{
  for(i in output)
    for(j in output[i])
      print j","i,output[i][j]
}

I find well named variables can also help eliminate the need for extensive comments

If i use your codes and change the input, like:

Code:

Sep 12 22:27:37 ipb named[3045]: error (connection refused) resolving 'wintoflashsuggestor.net/A/IN': 162.243.44.52#53
Sep 13 02:38:31 ipb named[3045]: client 183.56.172.145#20000: query (cache) '2054061883.www.baidu.com/A/IN' denied
Sep 18 22:31:20 ipb named[2677]: error (connection refused) resolving 'wintoflashsuggestor.net/A/IN': 162.243.44.52#53
Sep 18 23:53:02 ipb named[2677]: client 95.191.131.23#58068: query (cache) 'htyper.xyz/A/IN' denied
Sep 19 01:14:27 ipb named[2677]: client 119.97.137.184#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 01:19:09 ipb named[2677]: client 183.213.22.60#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 01:27:33 ipb named[2677]: client 218.60.5.146#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 01:39:20 ipb named[2677]: client 122.70.134.81#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 02:36:08 ipb named[2677]: client 95.215.60.214#42589: query (cache) 'defcon.org/RRSIG/IN' denied
Sep 19 02:37:57 ipb named[2677]: client 183.56.172.145#20000: query (cache) '2054061883.www.baidu.com/A/IN' denied
Sep 19 02:55:19 ipb named[2677]: client 113.17.184.25#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 03:23:46 ipb named[2677]: client 141.212.122.129#40036: query (cache) 'c.afekv.com/A/IN' denied
Sep 19 03:36:27 ipb named[2677]: client 119.97.137.184#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 03:39:22 ipb named[2677]: client 183.213.22.60#23456: query (cache) '998207098-43.com/A/IN' denied
Sep 19 05:09:47 ipb named[2677]: client 183.213.22.60#23457: query (cache) '94195337-3084195388-2359982150.ns.timely.dns-spider.ffdns.net/A/IN' denied
Sep 19 05:09:47 ipb named[2677]: client 183.213.22.60#23457: query (cache) '94195338-3084195388-2359982150.ns.timely.dns-spider.ffdns.net/A/IN' denied
Sep 19 06:31:04 ipb named[2677]: client 134.147.203.115#29267: query (cache) 'teg5.7a6e7f3b.wc.syssec.rub.de/A/IN' denied

output

Code:

, 1
119.97.137.184,query_denied 2
134.147.203.115,query_denied 1
122.70.134.81,query_denied 1
218.60.5.146,query_denied 1
183.213.22.60,query_denied 4
183.56.172.145,query_denied 2
113.17.184.25,query_denied 1
95.215.60.214,query_denied 1
141.212.122.129,query_denied 1
95.191.131.23,query_denied 1
134.147.203.11529267,query_denied 1
183.56.172.14520000,query_denied 1

How to fix?
Thanks.

I also try the long bash codes,like

Code:

#!/bin/bash
SAVE_IP=/allow_ip
TRUE_IP="([0-9]{1,3}[\.]){3}[0-9]{1,3}"
awk -F'[][()#=/,]|client |smtp:|submission:|unknown|host: ' '{for(z=1;z<=NF;z++) {\
	{/query.*denied/ && $z ~ /'"$TRUE_IP"'/						?a[$z]++:0;}
      	{/max connection count/ && $z ~ /'"$TRUE_IP"'/					?b[$z]++:0;}
      	{/SSL_accept error from unknown/ && !/submission/ && $z ~ /'"$TRUE_IP"'/	?c[$z]++:0;}
      	{/SSL_accept error from unknown/ && /submission/ && $z ~ /'"$TRUE_IP"'/		?s[$z]++:0;}
      	{/attackalert: Connect from host:/ && $z ~ /'"$TRUE_IP"'/			?d[$z]++:0;}
      	{/lost connection after CONNECT from unknown/ && !/\[unknown\]/ && $z ~ /'"$TRUE_IP"'/	?e[$z]++:0;}
      	{/Illegal address syntax from unknown/ && $z ~ /'"$TRUE_IP"'/			?f[$z]++:0;}
      	{/Disconnected: Inactivity \(no/ && $z ~ /'"$TRUE_IP"'/				?g[$z]++:0;}
      	{/Disconnected: Inactivity \(tried/ && $z ~ /'"$TRUE_IP"'/			?h[$z]++:0;}
      	{/Disconnected \(no auth attempts in/ && $z ~ /'"$TRUE_IP"'/			?i[$z]++:0;}
      	{/cannot find your reverse hostname/ && $z ~ /'"$TRUE_IP"'/			?j[$z]++:0;}
      	{/cannot find your hostname/ && $z ~ /'"$TRUE_IP"'/				?k[$z]++:0;}
      	{/Relay access denied/ && $z ~ /'"$TRUE_IP"'/					?l[$z]++:0;}
      	{/too many errors/ && $z ~ /'"$TRUE_IP"'/					?m[$z]++:0;}
      	{/Aborted login \(no/ && $z ~ /'"$TRUE_IP"'/					?n[$z]++:0;}
      	{/Aborted login \(tried/ && $z ~ /'"$TRUE_IP"'/					?o[$z]++:0;}
      	{/lost connection after AUTH from unknown/ && $z ~ /'"$TRUE_IP"'/		?p[$z]++:0;}
      	{/after AUTH from/ && !/unknown/ && $z ~ /'"$TRUE_IP"'/				?q[$z]++:0;}
      	{/client_address\=/ && $z ~ /'"$TRUE_IP"'/					?r[$z]++:0;}
	}}
END{
	for(y in a){if(a[y]>0) printf "%-15s %-27s %-2s\n",y,",query_denied",a[y]}
     	for(y in b){if(b[y]>0) printf "%-15s %-27s %-2s\n",y,",max_connection_count",b[y]}
     	for(y in c){if(c[y]>0) printf "%-15s %-27s %-2s\n",y,",SSL_accept_error",c[y]}
     	for(y in d){if(d[y]>0) printf "%-15s %-27s %-2s\n",y,",attackalert",d[y]}
     	for(y in e){if(e[y]>0) printf "%-15s %-27s %-2s\n",y,",lost_connection_after",e[y]}
     	for(y in f){if(f[y]>0) printf "%-15s %-27s %-2s\n",y,",Illegal_address",f[y]}
     	for(y in g){if(g[y]>0) printf "%-15s %-27s %-2s\n",y,",Disconnected_Inac_no",g[y]}
     	for(y in h){if(h[y]>0) printf "%-15s %-27s %-2s\n",y,",Disconnected_Inac_tried",h[y]}
     	for(y in i){if(i[y]>0) printf "%-15s %-27s %-2s\n",y,",Disconnected_no_auth",i[y]}
     	for(y in j){if(j[y]>0) printf "%-15s %-27s %-2s\n",y,",cannot_find_rev_host",j[y]}
     	for(y in k){if(k[y]>0) printf "%-15s %-27s %-2s\n",y,",cannot_find_your_host",k[y]}
     	for(y in l){if(l[y]>0) printf "%-15s %-27s %-2s\n",y,",Relay_access_denied",l[y]}
     	for(y in m){if(m[y]>0) printf "%-15s %-27s %-2s\n",y,",too_many_errors",m[y]}
     	for(y in n){if(n[y]>0) printf "%-15s %-27s %-2s\n",y,",Aborted_login_no",n[y]}
     	for(y in o){if(o[y]>0) printf "%-15s %-27s %-2s\n",y,",Aborted_login_tried",o[y]}
     	for(y in p){if(p[y]>0) printf "%-15s %-27s %-2s\n",y,",after_AUTH_from",p[y]}
     	for(y in q){if(q[y]>0) printf "%-15s %-27s %-2s\n",y,",after_AUTH_from_net",q[y]}
     	for(y in r){if(r[y]>0) printf "%-15s %-27s %-2s\n",y,",client_address",r[y]}
     	for(y in s){if(s[y]>0) printf "%-15s %-27s %-2s\n",y,",SSL_accept_error_sub",s[y]}
}' /abak/log/change_input | egrep -v "`cat $SAVE_IP | awk '{print $1}'`|port:|\.com|\.net|\.org"

And output is correct.

Code:

119.97.137.184  ,query_denied               2 
134.147.203.115 ,query_denied               1 
122.70.134.81   ,query_denied               1 
218.60.5.146    ,query_denied               1 
183.213.22.60   ,query_denied               4 
183.56.172.145  ,query_denied               2 
113.17.184.25   ,query_denied               1 
95.215.60.214   ,query_denied               1 
141.212.122.129 ,query_denied               1 
95.191.131.23   ,query_denied               1

I can change the match times for each pattern and no need to worry about where the field of IP is.

grail · 10-18-2016, 11:17 AM

The ip's with the additional port numbers on them are a by product of previous lines not being read properly because I was not catching lines we didn't care about and the final lines of the
original braces is always run. A simple default case fixed it for me:

Code:

#!/usr/bin/awk -f

{
	regex = "[^0-9.]"

	switch ($0){
		case /query/:
			if($NF == "denied"){
				field = $7
				name  = "query_denied"
				regex = "#.*"
			}
			break
		case /max connection/:
			field = $(NF-4)
			name  = "max_connection_count"
			break
		case /SSL_accept/:
			field = $(NF-2)
			name  = "SSL_accept_error"
			break
		default:
			name = "NOT TRACKED"
			ip   = "0.0.0.0"
	}

	if(name != "NOT TRACKED")
		ip = gensub(regex,"","g",field)

	output[name][ip]++
}

END{
	for(i in output)
		for(j in output[i])
		print j","i,output[i][j]
	}