[SOLVED] Filter through line/s to grab specific fields/data in the line with example

shayno90 · 10-13-2009, 09:11 AM

Hi, I am having a problem of getting specific data from fields in a line for example in this line (there are multiple lines like this just to let you know)

'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

I want to get just the 2 IP addresses at the end i.e. 208.67.220.220 and 95.224.96.106 but using this command (extra problem is I need the port numbers beside at the end each IP address i.e 208.67.220.220:53

cat filename | awk -F ":" '{ print $9 }'

i only get

{UDP} 208.67.220.220

however this commands works when there is an extra field like in this line

'MOP22','auth','info','info','26','2009-10-03 05:49:47','snort','snort[5193]: [1:402:8] ICMP Destination Unreachable Port Unreachable [Classification: Misc activity] [Priority: 3]: {ICMP} 86.47.191.211 -> 95.224.96.106',7285638

with same command i get

{ICMP} 83.237.90.142 -> 95.224.96.106',7150278

again all i need are the 2 IP addresses

any ideas anyone has would be appreciated!

forrestt · 10-13-2009, 09:38 AM

Try:

Code:

sed -e 's/.*}//' -e "s/\'.*//" -e 's/^ //' -e 's/-> //' filename

HTH

Forrest

p.s. Code is wrong. Fixed Below.

ComradeP · 10-13-2009, 09:41 AM

I'd suggest something like this:

Code:

cat filename | perl -n -e 'm/([0-9:.]+) -> ([0-9:.]+)/ and print "$1 -> $2\n";'

shayno90 · 10-13-2009, 11:42 AM

Thanks guys, did some playing around with the code

sed -e 's/.*{//' -e 's/^ //' -e 's/-> //' filename | tr -d "'" | sed -e 's/:/,/g' -e 's/ /,/g' -e 's/$.*$,.*/\1/' |tr -d "}" > filename.ods

and it prints this out

ICMP,192.168.200.1,95.224.96.106

which is the format i need to show it in an ods document

but is there a better way to shorten the code as it is quite long and hard to read?

forrestt · 10-13-2009, 12:05 PM

Above code I pasted should be:

Code:

sed -e 's/.*}//' -e "s/'.*//" -e 's/^ //' -e 's/-> //' filename

Forrest

ComradeP · 10-13-2009, 12:14 PM

This should do it:

Code:

cat filename | perl -n -e 'm/{(\w+)} ([0-9:.]+) -> ([0-9:.]+)/ and print "$1,$2,$3\n";'

forrestt · 10-13-2009, 12:16 PM

In your first post you said you wanted port numbers, then you post output w/o them. Is this because you want them but that line was ICMP or because you don't want them?

Forrest

PTrenholme · 10-13-2009, 01:22 PM

First, consider what using ":" as a field separator does:

Code:

$ echo 'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744 |\
gawk -F ":" '{for (i=1;i <=NF;++i) print i ": " $i;}
> '
1: MOP22,auth,info,info,26,2009-10-02 15
2: 13
3: 11,snort,snort[5193]
4:  [1
5: 254
6: 7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification
7:  Potentially Bad Traffic] [Priority
8:  2]
9:  {UDP} 208.67.220.220
10: 53 -> 95.224.96.106
11: 50543,717474

and consider this:

Code:

$ echo 'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744 |\
gawk -F ":" '{for (i=9;i <=NF;++i) s=s (s==""?"":":") $i;gsub(/[^0-9.:]+/," ",s);print s}'
 208.67.220.220:53 95.224.96.106:50543 7174744

which should give you a clew to solving your problem.

shayno90 · 10-14-2009, 03:47 AM

To answer your question, Forrest there are no ports associated with ICMP, only TCP and UDP. I only selected ICMP as an example and in relation to the revised I might try to add a comma as a separator in order to separate them in spreadsheet.. Also, to PTrenholme I don't have GAWK installed on my machine, just MAWK and when I ran your code it was like an infinite loop so I don't know if the probklem was that the code only functions properly with GAWK or there was an error in the code.

forrestt · 10-14-2009, 08:16 AM

I know there are no ports w/ ICMP, that's why I asked the question. Do you want the ports included if it is TCP or UDP (meaning that your example lacked them simply because it was an ICMP entry) or do you not want them for TCP and UDP as well (meaning your example lacked them because you don't want them at all)?

Forrest

shayno90 · 10-14-2009, 10:49 AM

Ya I want to keep the ports associated with TCP an UDP but since there are none with ICMP it is not necessary, here is the code I used to do so:

sed -e 's/.*{//' -e 's/} /,/' -e "s/'.*//" -e 's/-> //' -e 's/ /,/g' -e 's/:/,/g' filename > filename.ods

This commands prints out this before sending to an ods document

TCP,95.224.96.106,60207,64.4.9.31,1863
UDP,208.67.222.222,53,95.224.96.106,44237
ICMP,192.168.200.1,95.224.96.106

So the job is done

Any suggestions though for replacing the highlighted field below with a converted field?

'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

To be like this

'MOP22','auth','info','info','26','115688929','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

Tried this code but not sure how to correct it

cat filename | awk -F "," '{print $6}'|xargs -I $6 date -d $6 +"%s"

Thanks!

PTrenholme · 10-14-2009, 11:51 AM

Quote:

Originally Posted by shayno90

. . . Also, to PTrenholme I don't have GAWK installed on my machine, just MAWK and when I ran your code it was like an infinite loop so I don't know if the probklem was that the code only functions properly with GAWK or there was an error in the code.

According to the MAWK manual, that code (which is POSIX-standard AWK) should have run.

Try this:

Code:

{
# Skip all fields preceding the 9th ":"-delimited field
  FS=":"
  $0=$0                # Re-parse the input line
  if (NF < 9) next        # Skip if there are fewer than 9 fields
  for (i=9;i <=NF;++i) {
    s=s (s==""?"":":") $i
  }
  sub(/[^0-9.,:]+/," ",s)    # Replace any character not a digit, dot, colon, or comma by a blank 
  sub(/^ +/,"",s)         # Remove any leading blanks
# Re-parse using blank as a field delimiter
  FS=" "
  $0=s
# Output the result
  print "From line " NR ": " s
}

Run like this:

Code:

$ awk -f sep.awk sep.txt
From line 1: 208.67.220.220:53 -> 95.224.96.106:50543,7174744

Where that code is in sep.awk and the data file is in sep.txt. (Obviously, I have only your example line in my sep.txt.)

Note that this code:

Code:

BEGIN {
   FS=":"
}
{
# Skip all fields preceding the 9th ":"-delimited field
  if (NF < 9) next        # Skip if there are fewer than 9 fields
  for (i=9;i <=NF;++i) {
    s=s (s==""?"":":") $i
  }
  sub(/[^0-9.,:]+/," ",s)    # Replace any character not a digit, dot, colon, or comma by a blank 
  sub(/^ +/,"",s)         # Remove any leading blanks
# Output the result
  print "From line " NR ": " s
}

also works. I put the parse/re-parse and FS change stuff in the first code block to illustrate how you can dynamically change the field separator. You may find that technique useful in more sophisticated programs.