LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-12-2009, 11:07 PM   #1
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,546

Rep: Reputation: 50
A script that removes dubblets lines into a TXT file?


Hello,
Eventually, does this program already exist for bash/console ?

Thanks !
 
Old 06-13-2009, 06:16 AM   #2
j-ray
Senior Member
 
Registered: Jan 2002
Location: germany
Distribution: ubuntu
Posts: 1,413

Rep: Reputation: 99
if you put the following code in a separate file, call it "doubleFilter.pl" it will filter the file given in
$in to the file in $out. You should set 2 different files. If you want to overwrite the first file comment out the last line (remove the "#"). The new file will be free of direct double lines. (Is that what you wanted?)

#!/usr/bin/perl

$in="/home/me/perl/test.txt";
$out="/home/me/perl/test2.txt";
open(IN,"$in");
open(OUT,"> $out");
while(<IN>){
print OUT if ($_ ne $old);
$old = $_;
}
close(IN);
close(OUT);
#system("mv $out $in");
 
Old 06-13-2009, 06:38 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,050

Rep: Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971
"man uniq"
Depends if the OP really was only talking of 2 successive identical lines (rather than 3, or 4, or ...)
 
Old 06-13-2009, 07:30 AM   #4
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,276

Rep: Reputation: 170Reputation: 170
sort file | uniq -d > newfile
 
Old 06-13-2009, 08:15 AM   #5
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,453

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
This remove double lines without sorting:
Code:
awk '!($0 in a){a[$0]=1;print}' infile > oufile
 
Old 06-13-2009, 10:48 AM   #6
sieira
Member
 
Registered: Dec 2007
Location: Alcalá de Henares (Madrid)
Distribution: Debian
Posts: 40

Rep: Reputation: 16
It exist. Use dos2unix (unix2dos exists also)
 
Old 06-13-2009, 11:47 AM   #7
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
Quote:
Originally Posted by colucix View Post
This remove double lines without sorting:
Code:
awk '!($0 in a){a[$0]=1;print}' infile > oufile
Code:
awk '!a[$0]++' file
 
Old 06-13-2009, 12:53 PM   #8
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,453

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
Quote:
Originally Posted by ghostdog74 View Post
Code:
awk '!a[$0]++' file
Sgrunt!


 
Old 06-13-2009, 11:08 PM   #9
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,546

Original Poster
Rep: Reputation: 50
Wow, it's cool. It exists even an 'unique' function in Linux.

Thank you!!
 
Old 06-14-2009, 12:20 AM   #10
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
Quote:
Originally Posted by frenchn00b View Post
Wow, it's cool. It exists even an 'unique' function in Linux.

Thank you!!
hmm, after 1600+ posts and now you came to realize that?
 
Old 06-14-2009, 04:22 AM   #11
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,546

Original Poster
Rep: Reputation: 50
Quote:
Originally Posted by ghostdog74 View Post
hmm, after 1600+ posts and now you came to realize that?
well those simple functions, I never find them with apt-cache search.

like unix2dos, unique, ...
 
Old 06-14-2009, 06:02 AM   #12
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
its uniq, not unique. doing a listing of :
Code:
ls -l /usr/bin
 
Old 06-14-2009, 06:23 AM   #13
sieira
Member
 
Registered: Dec 2007
Location: Alcalá de Henares (Madrid)
Distribution: Debian
Posts: 40

Rep: Reputation: 16
Quote:
Originally Posted by ghostdog74 View Post
its uniq, not unique.
He ment unique, as "a single one"
 
Old 06-14-2009, 06:32 AM   #14
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
'apropos <searchterm>' (aka 'man -k <searchterm>') is a good way to search for useful commands. It will list all man pages related to the term you use.

And don't forget google.
 
Old 06-14-2009, 01:57 PM   #15
frenchn00b
Senior Member
 
Registered: Jun 2007
Location: E.U., Mountains :-)
Distribution: Debian, Etch, the greatest
Posts: 2,546

Original Poster
Rep: Reputation: 50
Quote:
Originally Posted by David the H. View Post
'apropos <searchterm>' (aka 'man -k <searchterm>') is a good way to search for useful commands. It will list all man pages related to the term you use.

And don't forget google.
Is there a databank/database for finding useful Bash/sh script?
like for video, mp3, ascii, txt, ... it does exists for visual basic, so why not for linux?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Bash script to compare numbers in a txt file leopard86 Programming 6 09-11-2012 12:10 AM
Finding things on lines of a txt file Jayfrin Programming 3 05-15-2008 08:13 AM
How can read from file.txt C++ where can save this file(file.txt) to start reading sam_22 Programming 1 01-11-2007 05:11 PM
shell script for changing a txt file content Flobsi Linux - Newbie 3 10-06-2006 03:10 AM
shell script for changig txt-file content Flobsi Programming 1 10-05-2006 05:46 PM


All times are GMT -5. The time now is 04:47 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration