LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-30-2013, 06:02 AM   #1
qrange
Member
 
Registered: Jul 2006
Location: Belgrade, Yugoslavia
Distribution: Debian stable, amd64
Posts: 805

Rep: Reputation: 32
comparing lines of two text files


I'm trying to check if all lines from one small text file (1Mb) are matched in a very very large text file (4Gb). both text files have the same format, so the lines should be identical.
Is there a fast and simple way of listing all lines that are not found in big file?
I had tried using perl, checking line by line, but its veeeery slow.
thanks.
 
Old 11-30-2013, 06:05 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
Have a look at the comm command:
Code:
comm -23 small_file big_file
EDIT: Both files need to be sorted first:
Code:
comm -23 <( sort small_file) <( sort big_file)

Last edited by druuna; 11-30-2013 at 06:08 AM. Reason: sorting the files is important....
 
3 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
AWK comparing 2 files and retrieving lines that have a matching field tekvaio Programming 1 11-27-2012 07:55 AM
Comparing lines in two files? needhelp12 Linux - Newbie 3 11-13-2011 12:48 AM
Comparing lines in 2 text files? Raveolution Linux - General 6 11-06-2010 04:59 PM
Shell script for comparing certain lines in two files mou5e Linux - Newbie 9 06-06-2007 02:40 PM
Comparing text files... jong357 Slackware 14 03-31-2007 05:29 PM


All times are GMT -5. The time now is 09:10 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration