[SOLVED] remove all text but IP addresses

frsechet · 11-06-2013, 03:07 AM

Hi,

I have a big log file with 20k lines with lots of info on it, but I want to only keep the IP addresses (and, if possible, deduplicated). For example, each line may read something like:

192.168.1.1 - lots of text that I do not want to keep [ZEFrgfosiglwkdnqbsg] GET /dnqfnqs

Then, after I'm left with only the IP, I would like to sort/deduplicate the lines.

The IPs are always at the beginning of each line.

Is there a command I could use for that?

Thanks!

shivaa · 11-06-2013, 03:15 AM

If IPs are always at the beginning of each line, then you can simply try:

Code:

~$ awk -F" " '{print $1}' logfile.txt

Or you can redirect the output to a new file, as:

Code:

~$ awk -F" " '{print $1}' logfile.txt > IPs.txt

evo2 · 11-06-2013, 03:17 AM

Hi,

it the IP is always the first thing in the line and it it always followed by a space, then the following onliner should work:

Code:

cut -f1 -d' ' inputfile | sort | uniq > outputfile

If the format it not that strict, it can still be done but we need more information about what is expected.

Evo2.

frsechet · 11-06-2013, 03:21 AM

That's exactly what I needed. The format was that strict...

Thanks!

grail · 11-06-2013, 06:55 AM

Code:

awk '!_[$1]++{print $1}' file | sort