LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 10-13-2009, 09:11 AM   #1
shayno90
Member
 
Registered: Oct 2009
Distribution: Debian Lenny 2.6.26 Ubuntu Lucid Lynx 10.04 Windows 7
Posts: 177
Blog Entries: 1

Rep: Reputation: 21
Filter through line/s to grab specific fields/data in the line with example


Hi, I am having a problem of getting specific data from fields in a line for example in this line (there are multiple lines like this just to let you know)

'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

I want to get just the 2 IP addresses at the end i.e. 208.67.220.220 and 95.224.96.106 but using this command (extra problem is I need the port numbers beside at the end each IP address i.e 208.67.220.220:53

cat filename | awk -F ":" '{ print $9 }'

i only get

{UDP} 208.67.220.220

however this commands works when there is an extra field like in this line

'MOP22','auth','info','info','26','2009-10-03 05:49:47','snort','snort[5193]: [1:402:8] ICMP Destination Unreachable Port Unreachable [Classification: Misc activity] [Priority: 3]: {ICMP} 86.47.191.211 -> 95.224.96.106',7285638

with same command i get

{ICMP} 83.237.90.142 -> 95.224.96.106',7150278

again all i need are the 2 IP addresses

any ideas anyone has would be appreciated!
 
Old 10-13-2009, 09:38 AM   #2
forrestt
Senior Member
 
Registered: Mar 2004
Location: Cary, NC, USA
Distribution: Fedora, Kubuntu, RedHat, CentOS, SuSe
Posts: 1,288

Rep: Reputation: 99
Try:

Code:
sed -e 's/.*}//' -e "s/\'.*//" -e 's/^ //' -e 's/-> //' filename
HTH

Forrest

p.s. Code is wrong. Fixed Below.

Last edited by forrestt; 10-13-2009 at 12:04 PM.
 
Old 10-13-2009, 09:41 AM   #3
ComradeP
LQ Newbie
 
Registered: Oct 2009
Posts: 12

Rep: Reputation: 2
I'd suggest something like this:

Code:
cat filename | perl -n -e 'm/([0-9:.]+) -> ([0-9:.]+)/ and print "$1 -> $2\n";'

Last edited by ComradeP; 10-13-2009 at 09:42 AM.
 
Old 10-13-2009, 11:42 AM   #4
shayno90
Member
 
Registered: Oct 2009
Distribution: Debian Lenny 2.6.26 Ubuntu Lucid Lynx 10.04 Windows 7
Posts: 177
Blog Entries: 1

Original Poster
Rep: Reputation: 21
Thanks guys, did some playing around with the code

sed -e 's/.*{//' -e 's/^ //' -e 's/-> //' filename | tr -d "'" | sed -e 's/:/,/g' -e 's/ /,/g' -e 's/\(.*\),.*/\1/' |tr -d "}" > filename.ods

and it prints this out

ICMP,192.168.200.1,95.224.96.106

which is the format i need to show it in an ods document

but is there a better way to shorten the code as it is quite long and hard to read?
 
Old 10-13-2009, 12:05 PM   #5
forrestt
Senior Member
 
Registered: Mar 2004
Location: Cary, NC, USA
Distribution: Fedora, Kubuntu, RedHat, CentOS, SuSe
Posts: 1,288

Rep: Reputation: 99
Above code I pasted should be:

Code:
sed -e 's/.*}//' -e "s/'.*//" -e 's/^ //' -e 's/-> //' filename
Forrest
 
Old 10-13-2009, 12:14 PM   #6
ComradeP
LQ Newbie
 
Registered: Oct 2009
Posts: 12

Rep: Reputation: 2
This should do it:

Code:
cat filename | perl -n -e 'm/{(\w+)} ([0-9:.]+) -> ([0-9:.]+)/ and print "$1,$2,$3\n";'
 
Old 10-13-2009, 12:16 PM   #7
forrestt
Senior Member
 
Registered: Mar 2004
Location: Cary, NC, USA
Distribution: Fedora, Kubuntu, RedHat, CentOS, SuSe
Posts: 1,288

Rep: Reputation: 99
In your first post you said you wanted port numbers, then you post output w/o them. Is this because you want them but that line was ICMP or because you don't want them?

Forrest
 
Old 10-13-2009, 01:22 PM   #8
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,147

Rep: Reputation: 330Reputation: 330Reputation: 330Reputation: 330
First, consider what using ":" as a field separator does:
Code:
$ echo 'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744 |\
gawk -F ":" '{for (i=1;i <=NF;++i) print i ": " $i;}
> '
1: MOP22,auth,info,info,26,2009-10-02 15
2: 13
3: 11,snort,snort[5193]
4:  [1
5: 254
6: 7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification
7:  Potentially Bad Traffic] [Priority
8:  2]
9:  {UDP} 208.67.220.220
10: 53 -> 95.224.96.106
11: 50543,717474
and consider this:
Code:
$ echo 'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744 |\
gawk -F ":" '{for (i=9;i <=NF;++i) s=s (s==""?"":":") $i;gsub(/[^0-9.:]+/," ",s);print s}'
 208.67.220.220:53 95.224.96.106:50543 7174744
which should give you a clew to solving your problem.
 
Old 10-14-2009, 03:47 AM   #9
shayno90
Member
 
Registered: Oct 2009
Distribution: Debian Lenny 2.6.26 Ubuntu Lucid Lynx 10.04 Windows 7
Posts: 177
Blog Entries: 1

Original Poster
Rep: Reputation: 21
To answer your question, Forrest there are no ports associated with ICMP, only TCP and UDP. I only selected ICMP as an example and in relation to the revised I might try to add a comma as a separator in order to separate them in spreadsheet.. Also, to PTrenholme I don't have GAWK installed on my machine, just MAWK and when I ran your code it was like an infinite loop so I don't know if the probklem was that the code only functions properly with GAWK or there was an error in the code.
 
Old 10-14-2009, 08:16 AM   #10
forrestt
Senior Member
 
Registered: Mar 2004
Location: Cary, NC, USA
Distribution: Fedora, Kubuntu, RedHat, CentOS, SuSe
Posts: 1,288

Rep: Reputation: 99
I know there are no ports w/ ICMP, that's why I asked the question. Do you want the ports included if it is TCP or UDP (meaning that your example lacked them simply because it was an ICMP entry) or do you not want them for TCP and UDP as well (meaning your example lacked them because you don't want them at all)?

Forrest
 
Old 10-14-2009, 10:49 AM   #11
shayno90
Member
 
Registered: Oct 2009
Distribution: Debian Lenny 2.6.26 Ubuntu Lucid Lynx 10.04 Windows 7
Posts: 177
Blog Entries: 1

Original Poster
Rep: Reputation: 21
Ya I want to keep the ports associated with TCP an UDP but since there are none with ICMP it is not necessary, here is the code I used to do so:

sed -e 's/.*{//' -e 's/} /,/' -e "s/'.*//" -e 's/-> //' -e 's/ /,/g' -e 's/:/,/g' filename > filename.ods

This commands prints out this before sending to an ods document

TCP,95.224.96.106,60207,64.4.9.31,1863
UDP,208.67.222.222,53,95.224.96.106,44237
ICMP,192.168.200.1,95.224.96.106

So the job is done

Any suggestions though for replacing the highlighted field below with a converted field?

'MOP22','auth','info','info','26','2009-10-02 15:13:11','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

To be like this

'MOP22','auth','info','info','26','115688929','snort','snort[5193]: [1:254:7] DNS SPOOF query response with TTL of 1 min. and no authority [Classification: Potentially Bad Traffic] [Priority: 2]: {UDP} 208.67.220.220:53 -> 95.224.96.106:50543',7174744

Tried this code but not sure how to correct it

cat filename | awk -F "," '{print $6}'|xargs -I $6 date -d $6 +"%s"

Thanks!
 
Old 10-14-2009, 11:51 AM   #12
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,147

Rep: Reputation: 330Reputation: 330Reputation: 330Reputation: 330
Quote:
Originally Posted by shayno90 View Post
. . . Also, to PTrenholme I don't have GAWK installed on my machine, just MAWK and when I ran your code it was like an infinite loop so I don't know if the probklem was that the code only functions properly with GAWK or there was an error in the code.
According to the MAWK manual, that code (which is POSIX-standard AWK) should have run.

Try this:
Code:
{
# Skip all fields preceding the 9th ":"-delimited field
  FS=":"
  $0=$0                # Re-parse the input line
  if (NF < 9) next        # Skip if there are fewer than 9 fields
  for (i=9;i <=NF;++i) {
    s=s (s==""?"":":") $i
  }
  sub(/[^0-9.,:]+/," ",s)    # Replace any character not a digit, dot, colon, or comma by a blank 
  sub(/^ +/,"",s)         # Remove any leading blanks
# Re-parse using blank as a field delimiter
  FS=" "
  $0=s
# Output the result
  print "From line " NR ": " s
}
Run like this:
Code:
$ awk -f sep.awk sep.txt
From line 1: 208.67.220.220:53 -> 95.224.96.106:50543,7174744
Where that code is in sep.awk and the data file is in sep.txt. (Obviously, I have only your example line in my sep.txt.)

Note that this code:
Code:
BEGIN {
   FS=":"
}
{
# Skip all fields preceding the 9th ":"-delimited field
  if (NF < 9) next        # Skip if there are fewer than 9 fields
  for (i=9;i <=NF;++i) {
    s=s (s==""?"":":") $i
  }
  sub(/[^0-9.,:]+/," ",s)    # Replace any character not a digit, dot, colon, or comma by a blank 
  sub(/^ +/,"",s)         # Remove any leading blanks
# Output the result
  print "From line " NR ": " s
}
also works. I put the parse/re-parse and FS change stuff in the first code block to illustrate how you can dynamically change the field separator. You may find that technique useful in more sophisticated programs.
 
  


Reply

Tags
extract, fields, line, same, separate


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sed append text to end of line if line contains specific text? How can this be done? helptonewbie Linux - Newbie 4 10-23-2013 01:48 PM
[SOLVED] specifying fields for printing in gawk from command line David the H. Programming 8 08-04-2009 03:32 PM
grab the line below a blank line and the line above the next blank line awk or perl? Pantomime Linux - General 7 06-26-2008 08:13 AM
php - Read file line by line and change a specific line. anrea Programming 2 01-28-2007 01:43 PM
How to use awk command to parse fields in a line johnsanty Programming 9 05-25-2006 09:56 PM


All times are GMT -5. The time now is 10:03 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration