How to extract difference between two files?

Marshalle · 05-01-2005, 03:39 AM

Hey there, I'm hoping someone can help me. I have a large collection of MP3's taken from my own CD's and I decided to format my NTFS drive as Reiser (I have Suse 9.2 Pro). I have backed up many of them and got lost somewhere along the way. So my problem is how do I compile a list of songs not backed up to dvd for later burning. So far I have exported the individual DVD's to a text file and also the original list to a text file. I figured I could use diff to find out the differences, but the problem is I need to export all files that are not the same and can't quite figure it out? IE the files on the DVD should be taken away from the master list. Any ideas? I've never actually had a response before so lets see if I get some luck!

Thanks,

M

rjlee · 05-01-2005, 06:37 AM

diff keeps the information from both files to allow rolling back, whereas you want the actual set difference. I would do this with a little Perl script:

Code:

#!/usr/bin/perl
open a, "<filea";
open b, "<fileb";
local $/;
my @a = split /\n/, <a>;
my @b = split /\n/, <b>;
my %b = map { $_ => 1 } @b; # Make hash of B
my @res = grep { !defined $b{$_} } @a; # Everything in A not in B
print join "\n", @res;
print "\n";

I'm assuming that both files have one filename per line, with no newlines in filenames, and the same filename is rendered the same way in both files.

Replace filea with the file containing the list of all files, and fileb with the list of extracted files.
Go to the directory where the files are, then type

Code:

perl > diff.lst

Paste in the entire script, including the newline at the end, then press Ctrl+D.

Your difference file will be produced in diff.lst

Marshalle · 05-02-2005, 06:38 AM

Well thanks for the reply! It's 11:30 so I'm off to sleep but I had a quick try but it seemed to produce all of the contents of both of the files, probably something I did so will try again tomorrow when I'm actually half awake! Would it be easy to list multiple txt files in place of you fileb to save me having to keep reapplying them to a new master file? Looks like perl is something I should learn sometime! Thanks again!

M

rjlee · 05-03-2005, 03:04 AM

To remove the contents of multiple files, replace this:

Code:

open b, "<fileb";

with this:

Code:

open b, "cat fileb filec filed filee |";

(where | is a pipe character). You may need to shell-escape the filenames.

Marshalle · 05-03-2005, 04:10 AM

Well I put both of these examples together and they worked perfectly! I can't believe how fast it was either, I didn't even see the HDD light blink. So I sent a donation in, was soooo pleased. Thanks a million, great to know support like this is out there, now I can install this for other people with confidence.

burf · 10-01-2009, 03:02 AM

Thanks for the reply, it worked for me too

A great way to find the difference in two files

chrism01 · 10-02-2009, 12:38 AM

If you want to take up Perl ( I recommend it), here's a couple of good links:
http://perldoc.perl.org/
http://www.perlmonks.org/?node=Tutorials

alessandra86 · 01-04-2011, 03:06 PM

Hi!! I am new at linux and I was trying to do the same thing of this forum with my files, but it didn't work. I hope someone can help me.

I use the script and it work ok with a little example that I did (two files with shared numbers), but if I use my files, I dont get differences between them.

I don't know, but maybe is my file content. It has long lines with numbers and letters, like this:

F6U9T4101A8GYD
F6U9T4101BXUMB
F6U9T4101A23JN
F6U9T4101BVP37
F6U9T4101BU6FA
F6U9T4101BN8YQ
F6U9T4101ARLNS
F6U9T4101BYIWG

I got two files. One with the total number of ids (44200) and the other with 5500 ids. So I need the differences, so I would have 38700 ids in the diff file. But I don know what happend, I hope you can help me!!!

Regards, alessa

rjlee · 01-05-2011, 12:45 PM

alessandra86, you don't say what you actually did get. But the script returns everything in the first file that isn't in the second, so you didn't get anything then you might try again with the files the other way around…