LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-12-2009, 11:07 PM   #1
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,561

Rep: Reputation: 57
A script that removes dubblets lines into a TXT file?


Hello,
Eventually, does this program already exist for bash/console ?

Thanks !
 
Old 06-13-2009, 06:16 AM   #2
j-ray
Senior Member
 
Registered: Jan 2002
Location: germany
Distribution: ubuntu, mint, suse
Posts: 1,591

Rep: Reputation: 145Reputation: 145
if you put the following code in a separate file, call it "doubleFilter.pl" it will filter the file given in
$in to the file in $out. You should set 2 different files. If you want to overwrite the first file comment out the last line (remove the "#"). The new file will be free of direct double lines. (Is that what you wanted?)

#!/usr/bin/perl

$in="/home/me/perl/test.txt";
$out="/home/me/perl/test2.txt";
open(IN,"$in");
open(OUT,"> $out");
while(<IN>){
print OUT if ($_ ne $old);
$old = $_;
}
close(IN);
close(OUT);
#system("mv $out $in");
 
Old 06-13-2009, 06:38 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,126

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
"man uniq"
Depends if the OP really was only talking of 2 successive identical lines (rather than 3, or 4, or ...)
 
Old 06-13-2009, 07:30 AM   #4
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Mint, Armbian, NetBSD, Puppy, Raspbian
Posts: 3,515

Rep: Reputation: 239Reputation: 239Reputation: 239
sort file | uniq -d > newfile
 
Old 06-13-2009, 08:15 AM   #5
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
This remove double lines without sorting:
Code:
awk '!($0 in a){a[$0]=1;print}' infile > oufile
 
Old 06-13-2009, 10:48 AM   #6
sieira
Member
 
Registered: Dec 2007
Location: Alcalá de Henares (Madrid)
Distribution: Debian
Posts: 40

Rep: Reputation: 16
It exist. Use dos2unix (unix2dos exists also)
 
Old 06-13-2009, 11:47 AM   #7
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Quote:
Originally Posted by colucix View Post
This remove double lines without sorting:
Code:
awk '!($0 in a){a[$0]=1;print}' infile > oufile
Code:
awk '!a[$0]++' file
 
Old 06-13-2009, 12:53 PM   #8
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Quote:
Originally Posted by ghostdog74 View Post
Code:
awk '!a[$0]++' file
Sgrunt!


 
Old 06-13-2009, 11:08 PM   #9
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,561

Original Poster
Rep: Reputation: 57
Wow, it's cool. It exists even an 'unique' function in Linux.

Thank you!!
 
Old 06-14-2009, 12:20 AM   #10
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Quote:
Originally Posted by frenchn00b View Post
Wow, it's cool. It exists even an 'unique' function in Linux.

Thank you!!
hmm, after 1600+ posts and now you came to realize that?
 
Old 06-14-2009, 04:22 AM   #11
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,561

Original Poster
Rep: Reputation: 57
Quote:
Originally Posted by ghostdog74 View Post
hmm, after 1600+ posts and now you came to realize that?
well those simple functions, I never find them with apt-cache search.

like unix2dos, unique, ...
 
Old 06-14-2009, 06:02 AM   #12
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
its uniq, not unique. doing a listing of :
Code:
ls -l /usr/bin
 
Old 06-14-2009, 06:23 AM   #13
sieira
Member
 
Registered: Dec 2007
Location: Alcalá de Henares (Madrid)
Distribution: Debian
Posts: 40

Rep: Reputation: 16
Quote:
Originally Posted by ghostdog74 View Post
its uniq, not unique.
He ment unique, as "a single one"
 
Old 06-14-2009, 06:32 AM   #14
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
'apropos <searchterm>' (aka 'man -k <searchterm>') is a good way to search for useful commands. It will list all man pages related to the term you use.

And don't forget google.
 
Old 06-14-2009, 01:57 PM   #15
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,561

Original Poster
Rep: Reputation: 57
Quote:
Originally Posted by David the H. View Post
'apropos <searchterm>' (aka 'man -k <searchterm>') is a good way to search for useful commands. It will list all man pages related to the term you use.

And don't forget google.
Is there a databank/database for finding useful Bash/sh script?
like for video, mp3, ascii, txt, ... it does exists for visual basic, so why not for linux?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Bash script to compare numbers in a txt file leopard86 Programming 6 09-11-2012 12:10 AM
Finding things on lines of a txt file Jayfrin Programming 3 05-15-2008 08:13 AM
How can read from file.txt C++ where can save this file(file.txt) to start reading sam_22 Programming 1 01-11-2007 05:11 PM
shell script for changing a txt file content Flobsi Linux - Newbie 3 10-06-2006 03:10 AM
shell script for changig txt-file content Flobsi Programming 1 10-05-2006 05:46 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration