LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-06-2013, 04:07 AM   #1
frsechet
Member
 
Registered: May 2013
Posts: 32

Rep: Reputation: Disabled
remove all text but IP addresses


Hi,

I have a big log file with 20k lines with lots of info on it, but I want to only keep the IP addresses (and, if possible, deduplicated). For example, each line may read something like:

192.168.1.1 - lots of text that I do not want to keep [ZEFrgfosiglwkdnqbsg] GET /dnqfnqs

Then, after I'm left with only the IP, I would like to sort/deduplicate the lines.

The IPs are always at the beginning of each line.

Is there a command I could use for that?

Thanks!
 
Old 11-06-2013, 04:15 AM   #2
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
If IPs are always at the beginning of each line, then you can simply try:
Code:
~$ awk -F" " '{print $1}' logfile.txt
Or you can redirect the output to a new file, as:
Code:
~$ awk -F" " '{print $1}' logfile.txt > IPs.txt

Last edited by shivaa; 11-06-2013 at 04:17 AM.
 
Old 11-06-2013, 04:17 AM   #3
evo2
LQ Guru
 
Registered: Jan 2009
Location: Japan
Distribution: Mostly Debian and Scientific Linux
Posts: 5,753

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Hi,

it the IP is always the first thing in the line and it it always followed by a space, then the following onliner should work:
Code:
cut -f1 -d' ' inputfile | sort | uniq > outputfile
If the format it not that strict, it can still be done but we need more information about what is expected.

Evo2.
 
1 members found this post helpful.
Old 11-06-2013, 04:21 AM   #4
frsechet
Member
 
Registered: May 2013
Posts: 32

Original Poster
Rep: Reputation: Disabled
That's exactly what I needed. The format was that strict...

Thanks!
 
Old 11-06-2013, 07:55 AM   #5
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,424

Rep: Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823
Code:
awk '!_[$1]++{print $1}' file | sort
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
i need to write a c code to ping IP addresses which are read from a text file vinothtitan Linux - Newbie 5 10-13-2014 12:23 AM
Program addresses from text editor to ELF file. theKbStockpiler Linux - General 2 10-20-2011 03:04 AM
Remove Thunderbird's Faulty Addresses alaios Linux - Software 2 05-15-2010 11:08 AM
LDAP mail addresses dump to text otisthegbs Linux - Server 1 02-28-2008 08:41 PM
using sed to remove all but ip addresses chess Programming 10 07-02-2007 02:54 AM


All times are GMT -5. The time now is 12:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration