LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 07-01-2007, 11:09 AM   #1
chess
Member
 
Registered: Mar 2002
Location: 127.0.0.1
Distribution: Slackware, OpenBSD, FreeBSD
Posts: 728

Rep: Reputation: 168Reputation: 168
using sed to remove all but ip addresses


I am trying to extract out a list of ip addresses from a firewall log, with one ip on a line. So far I have come up with this:

cat log.txt > sed 's/^.*[0-9]\{1,3\}.[0-9]\{1,3\}.[0-9]\{1,3\}.[0-9]\{1,3\}//g'

but that's not working. If someone could help me, I would be very grateful. Thanks.
 
Old 07-01-2007, 11:35 AM   #2
kscott121
Member
 
Registered: Jul 2003
Location: NC
Distribution: Fedora,Mepis,Debian
Posts: 84

Rep: Reputation: 15
Include a little snippet of your log file so everybody is on the same page.
Ken
 
Old 07-01-2007, 11:41 AM   #3
chess
Member
 
Registered: Mar 2002
Location: 127.0.0.1
Distribution: Slackware, OpenBSD, FreeBSD
Posts: 728

Original Poster
Rep: Reputation: 168Reputation: 168
Ok, thanks, here goes:

Code:
Jul  1 12:35:01 puffy pf: Jul 01 12:32:24.830687 rule 0/(match) block in on sis0: 129.42.58.103.80 > 111.22.33.44.64702: F 0:0(0) ack 1 win 8190 [tos 0x60] 
Jul  1 12:35:01 puffy pf: Jul 01 12:32:24.830831 rule 0/(match) block in on sis0: 129.42.58.103.80 > 111.22.33.44.50012: F 0:0(0) ack 1 win 8190 [tos 0x60] 
Jul  1 12:35:01 puffy pf: Jul 01 12:33:54.090446 rule 0/(match) block in on sis0: 221.209.110.50.49214 > 111.22.33.44.1027: udp 457 (DF)
The "111.22.33.44" is my own ip, changed for the purposes of posting here. Basically, I am trying to extract out the incoming ips, in this case the 129.42.58.103 and 221.209.110.50 ip addresses, into a log file so I can see the occurrences.
 
Old 07-01-2007, 11:49 AM   #4
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: Ubuntu, FreeBSD, NetBSD
Posts: 1,449

Rep: Reputation: 47
Code:
awk '{print $15}' /path/to/logfile | awk -F. '{print $1"."$2"."$3"."$4}'
It is so much easier to use awk here... instead of sed. Note: the second half isn't even needed... except that I use it to chop off the port number.
 
Old 07-01-2007, 12:08 PM   #5
chess
Member
 
Registered: Mar 2002
Location: 127.0.0.1
Distribution: Slackware, OpenBSD, FreeBSD
Posts: 728

Original Poster
Rep: Reputation: 168Reputation: 168
That works great, thanks frob23!
 
Old 07-01-2007, 06:47 PM   #6
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
there's no need to use awk 2 times
Code:
awk '
     { n=split($15,ip,".")
	   r=ip[1]	   
       for(i=2;i<n;i++){
	     r=r"."ip[i] 
	   }
	   print r
     }
' "file"
 
Old 07-01-2007, 07:00 PM   #7
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

Or, using multiple field separators:
Code:
awk -F"[ .]" '{ print $17"."$18"."$19"."$20 }' infile
There's a space and a dot inside the square brackets.
 
Old 07-01-2007, 08:29 PM   #8
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,356

Rep: Reputation: 1043Reputation: 1043Reputation: 1043Reputation: 1043Reputation: 1043Reputation: 1043Reputation: 1043Reputation: 1043
Nice answers, but I'm inclined to parse for known data rather than rely on position.
Ya never know when the format will change ...

As always, each to their own - I'd probably do it in perl.
 
Old 07-01-2007, 08:48 PM   #9
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
well, given the sample input and from what i see, the only place that formatting can change is after the destination ip portion..so we can safely assuming fields 1 to 17 is kind of fixed. However, this is only a very small sample though..
 
Old 07-01-2007, 10:58 PM   #10
chess
Member
 
Registered: Mar 2002
Location: 127.0.0.1
Distribution: Slackware, OpenBSD, FreeBSD
Posts: 728

Original Poster
Rep: Reputation: 168Reputation: 168
Here is a longer sampling:

Code:
Jul  1 23:00:02 puffy pf: Jul 01 22:57:28.831284 rule 0/(match) block in on sis0
: 198.26.188.185.31169 > 111.22.33.44.1026: udp 373 
Jul  1 23:00:02 puffy pf: Jul 01 22:58:10.370293 rule 0/(match) block in on sis0
: 159.46.97.65.31169 > 111.22.33.44.1026: udp 493 
Jul  1 23:00:02 puffy pf: Jul 01 22:58:10.370472 rule 0/(match) block in on sis0
: 159.46.97.65.31169 > 111.22.33.44.1027: udp 493 
Jul  1 23:05:02 puffy pf: Jul 01 23:00:41.681482 rule 0/(match) block in on sis0
: 218.234.41.8.6000 > 111.22.33.44.6588: S 1851981824:1851981824(0) win 16384 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.790557 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.790837 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.791129 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1028: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.131783 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.132086 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1028: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.132375 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.492680 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.493063 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.500104 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1028: udp 484 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:05.750283 rule 0/(match) block in on sis0
: 24.121.217.166.1895 > 111.22.33.44.2968: S 3727943993:3727943993(0) win 1638
4 <mss 1460,nop,nop,sackOK> (DF) 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:08.750281 rule 0/(match) block in on sis0
: 24.121.217.166.1895 > 111.22.33.44.2968: S 3727943993:3727943993(0) win 1638
4 <mss 1460,nop,nop,sackOK> (DF) 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:46.380275 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 [ttl 1] 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:51.070190 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 [ttl 1] 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:56.040218 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 
Jul  1 23:15:02 puffy pf: Jul 01 23:13:01.050217 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4
All of your suggestions have been great, thanks everyone. Just in case anyone can't tell or is curious, this is an OpenBSD install on a Soekris 4801 running off a compact flash card. It's working great. Anyway, I have the regular pflog run through a script that does a tcpdump into a text file, which is what you see above. I have been watching these logs and for some reason there is an inordinate number of hits from various IPs all within the Shaw Communications range of 24.64.0.0 - 24.71.255.255. I googled around and found a forum thread at the zonealarm forums where other people have found similar hits. (http://forums.zonealarm.com/zonelabs...ssage.id=16151) . That thread starts off about Firefox but then discusses this Shaw Communications issue. I guess there are a lot of Shaw customers with infected computers.
 
Old 07-02-2007, 01:54 AM   #11
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,301

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Whichever ISP you're with, you'll always get a large amt of hits from within that (ie the ISP's) range.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SED - remove last four characters from string 3saul Linux - Software 5 07-28-2014 06:25 AM
help with sed to remove all text except for some Benanzo Linux - Software 7 01-04-2007 06:21 AM
how to remove path using sed dtcs Programming 2 12-25-2006 04:29 PM
Remove string in sed twantrd Programming 7 09-13-2006 02:28 PM
Bulk replace email addresses with SED ? wally.nl Programming 1 11-01-2005 10:42 AM


All times are GMT -5. The time now is 12:27 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration