Looking for directory tree comparison tool

joeljkp · 02-20-2007, 08:44 PM

I did a server move, and I want to make sure everything transferred over and wasn't corrupted. So I'm looking for some kind of directory tree comparison tool that will work over ssh between two remote machines (or that I can run on one in relation to the other).

I've tried downloading the results of `ls -lR` from both servers and diffed the resulting files, but this flagged every little change in modification times, etc., so it was useless to me.

I also tried rsync with -n (dry-run), but this seemed to just produce a list of files it would compare, without actually doing any comparison.

I've also noticed a couple tools out there just for this, but they both require Java, and neither of the servers has Java on it.

Any suggestions?

zulfilee · 02-21-2007, 01:11 AM

Go for a md5sum of the files and do a compare.

A simple loop will be [From the directory where u have all the files]

for EACHFILE in `find . -iname "*"`
do
md5sum -b $EACHFILE
done

Get the results from both the serves and compare

- zulfi

Quigi · 02-21-2007, 12:11 PM

Quote:

Originally Posted by zulfilee

for EACHFILE in `find . -iname "*"`

Hi Zulfilee,
What's the purpose of the -iname "*" test? It seems to succeed always.

---

BTW, md5sum doesn't like to operate on directories, so -type f might be useful.

Passing all files to md5sum should be a bit more efficient than starting a new process for each file. So:

Code:

md5sum `find . -type f`

Or, if you have so many files that that gives you a too long argument list (or weird characters in file names),

Code:

find . -type f -print0 | xargs -0 md5sum

You only need NUL delimiters (purple stuff) if you have weird characters in your file names, e.g., spaces.

zulfilee · 02-22-2007, 12:19 AM

Yeah the 'type' option will be proper .

-iname "*" tries to get all files and directories [which is not needed].

joeljkp · 02-22-2007, 10:46 AM

All of these solutions have a flaw if there is a possibility of there being additional or missing files. When using diff to compare the two sets of md5sums, it can't differentiate between items from different directories.

Consider the following:

Code:

Tree 1:
dir1 \
  file1
  file2
  file2~ (for example)
  file3

Tree 2:
dir1 \
  file1
  file2
  file3

diff:
<md5sum> dir1/file1   <md5sum> dir1/file1 <- match
<md5sum> dir1/file2   <md5sum> dir1/file2 <- match
<md5sum> dir1/file2~  <md5sum> dir1/file3 <- no match!
<md5sum> dir1/file3   <md5sum> dir2/file1 <- no match!
<md5sum> dir2/file1   <md5sum> dir2/file2 <- no match!

etc.

There would need to be something that has a concept of added and removed files, directories, etc.

Quigi · 02-22-2007, 12:44 PM

Quote:

Originally Posted by joeljkp

When using diff to compare the two sets of md5sums, it can't differentiate between items from different directories.

Not sure if I understand the part "from different directories", so I'll reply to the example.

Diff is smarter than that; it re-synchronizes.
I recreated your scenario. As we have a reasonable number of files with sensible names, the simple md5sum `find dir1 -type f` is enough. The respective outputs are:

Code:

b026324c6904b2a9cb4b88d6d61c81d1  dir1/file1
26ab0db90d72e28ad0ba1e22ee510510  dir1/file2
6d7fce9fee471194aa8b5b6e47267f03  dir1/file3

and

Code:

b026324c6904b2a9cb4b88d6d61c81d1  dir1/file1
26ab0db90d72e28ad0ba1e22ee510510  dir1/file2
c193497a1a06b2c72230e6146ff47080  dir1/file2~
6d7fce9fee471194aa8b5b6e47267f03  dir1/file3

And when I run diff on them, I get

Code:

2a3
> c193497a1a06b2c72230e6146ff47080  dir1/file2~

Note: my find (GNU find version 4.2.27) outputs the file names without sorting, i.e., like ls -f does. The files may be ordered differently between the two trees, which would cause spurious differences. If that's an issue, try md5sum `find dir1 -type f | sort` to get a predictable order.

BTW, initially you said

Quote:

I want to make sure everything transferred over and wasn't corrupted.

If diff is quiet, everything is well. If it reports any difference, the trees don't have identical content.