using sed to remove all but ip addresses

chess · 07-01-2007, 11:09 AM

I am trying to extract out a list of ip addresses from a firewall log, with one ip on a line. So far I have come up with this:

cat log.txt > sed 's/^.*[0-9]\{1,3\}.[0-9]\{1,3\}.[0-9]\{1,3\}.[0-9]\{1,3\}//g'

but that's not working. If someone could help me, I would be very grateful. Thanks.

kscott121 · 07-01-2007, 11:35 AM

Include a little snippet of your log file so everybody is on the same page.
Ken

chess · 07-01-2007, 11:41 AM

Ok, thanks, here goes:

Code:

Jul  1 12:35:01 puffy pf: Jul 01 12:32:24.830687 rule 0/(match) block in on sis0: 129.42.58.103.80 > 111.22.33.44.64702: F 0:0(0) ack 1 win 8190 [tos 0x60] 
Jul  1 12:35:01 puffy pf: Jul 01 12:32:24.830831 rule 0/(match) block in on sis0: 129.42.58.103.80 > 111.22.33.44.50012: F 0:0(0) ack 1 win 8190 [tos 0x60] 
Jul  1 12:35:01 puffy pf: Jul 01 12:33:54.090446 rule 0/(match) block in on sis0: 221.209.110.50.49214 > 111.22.33.44.1027: udp 457 (DF)

The "111.22.33.44" is my own ip, changed for the purposes of posting here. Basically, I am trying to extract out the incoming ips, in this case the 129.42.58.103 and 221.209.110.50 ip addresses, into a log file so I can see the occurrences.

frob23 · 07-01-2007, 11:49 AM

Code:

awk '{print $15}' /path/to/logfile | awk -F. '{print $1"."$2"."$3"."$4}'

It is so much easier to use awk here... instead of sed. Note: the second half isn't even needed... except that I use it to chop off the port number.

chess · 07-01-2007, 12:08 PM

That works great, thanks frob23!

ghostdog74 · 07-01-2007, 06:47 PM

there's no need to use awk 2 times

Code:

awk '
     { n=split($15,ip,".")
	   r=ip[1]	   
       for(i=2;i<n;i++){
	     r=r"."ip[i] 
	   }
	   print r
     }
' "file"

druuna · 07-01-2007, 07:00 PM

Hi,

Or, using multiple field separators:

Code:

awk -F"[ .]" '{ print $17"."$18"."$19"."$20 }' infile

There's a space and a dot inside the square brackets.

syg00 · 07-01-2007, 08:29 PM

Nice answers, but I'm inclined to parse for known data rather than rely on position.
Ya never know when the format will change ...

As always, each to their own - I'd probably do it in perl.

ghostdog74 · 07-01-2007, 08:48 PM

well, given the sample input and from what i see, the only place that formatting can change is after the destination ip portion..so we can safely assuming fields 1 to 17 is kind of fixed. However, this is only a very small sample though..

chess · 07-01-2007, 10:58 PM

Here is a longer sampling:

Code:

Jul  1 23:00:02 puffy pf: Jul 01 22:57:28.831284 rule 0/(match) block in on sis0
: 198.26.188.185.31169 > 111.22.33.44.1026: udp 373 
Jul  1 23:00:02 puffy pf: Jul 01 22:58:10.370293 rule 0/(match) block in on sis0
: 159.46.97.65.31169 > 111.22.33.44.1026: udp 493 
Jul  1 23:00:02 puffy pf: Jul 01 22:58:10.370472 rule 0/(match) block in on sis0
: 159.46.97.65.31169 > 111.22.33.44.1027: udp 493 
Jul  1 23:05:02 puffy pf: Jul 01 23:00:41.681482 rule 0/(match) block in on sis0
: 218.234.41.8.6000 > 111.22.33.44.6588: S 1851981824:1851981824(0) win 16384 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.790557 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.790837 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:05:08.791129 rule 0/(match) block in on sis0
: 24.64.70.164.32015 > 111.22.33.44.1028: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.131783 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.132086 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1028: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:03.132375 rule 0/(match) block in on sis0
: 24.64.170.225.10492 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.492680 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1026: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.493063 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1027: udp 484 
Jul  1 23:10:02 puffy pf: Jul 01 23:07:42.500104 rule 0/(match) block in on sis0
: 24.64.54.41.28243 > 111.22.33.44.1028: udp 484 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:05.750283 rule 0/(match) block in on sis0
: 24.121.217.166.1895 > 111.22.33.44.2968: S 3727943993:3727943993(0) win 1638
4 <mss 1460,nop,nop,sackOK> (DF) 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:08.750281 rule 0/(match) block in on sis0
: 24.121.217.166.1895 > 111.22.33.44.2968: S 3727943993:3727943993(0) win 1638
4 <mss 1460,nop,nop,sackOK> (DF) 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:46.380275 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 [ttl 1] 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:51.070190 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 [ttl 1] 
Jul  1 23:15:02 puffy pf: Jul 01 23:12:56.040218 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4 
Jul  1 23:15:02 puffy pf: Jul 01 23:13:01.050217 rule 0/(match) block in on sis0
: 63.251.178.30.13223 > 111.22.33.44.33436: udp 4

All of your suggestions have been great, thanks everyone. Just in case anyone can't tell or is curious, this is an OpenBSD install on a Soekris 4801 running off a compact flash card. It's working great. Anyway, I have the regular pflog run through a script that does a tcpdump into a text file, which is what you see above. I have been watching these logs and for some reason there is an inordinate number of hits from various IPs all within the Shaw Communications range of 24.64.0.0 - 24.71.255.255. I googled around and found a forum thread at the zonealarm forums where other people have found similar hits. (http://forums.zonealarm.com/zonelabs...ssage.id=16151) . That thread starts off about Firefox but then discusses this Shaw Communications issue. I guess there are a lot of Shaw customers with infected computers.

chrism01 · 07-02-2007, 01:54 AM

Whichever ISP you're with, you'll always get a large amt of hits from within that (ie the ISP's) range.