I've been saving lots of data from one of my NAS boxes to another one and that's been going fine...I thought. I recently found out that some of the data on my secondary NAS is different. I'm wondering what some of you might suggest about doing a mass file check of my mirror copy. Would you use diff or would you create and checksum and compare?

My directory structure looks something like this:

I've got hundreds of these directories with data files in them. Any recommondations of a script or something I should run to check them all and tell me which are different?

As always, thanks for the help.

rsync is a good choice. If you set it up to sync good copy to bad, it will only transfer the files that are different.

rsync docs:
http://samba.anu.edu.au/rsync/documentation.html

"diff" is only good on text files, I believe. It may have support for binary files with an option.
You could also look at "cmp", for binary comparisons.
However, both "diff" and "cmp" practically read and compare each block of each file, so you may be in for a long wait.
Using checksums, like md5s, is a very reliable alternative and much faster.
Check out:
man md5sum
(don't forget the -b option for binary files and to capture all output in a file. Reporting on screen
will only be time consuming.

But, as bulliver already pointed out, rsync or some other intelligent copying
software may be your friend too.

Thanks for the responses...

So even though I have two mounted NAS drives (/NAS1 and /NAS2), rsync is still a good way to go? Please excuse my ignorance. I thought that rsync was just for remote coping. Also, if I start using rsync, is there a switch in there that will tell it to compare checksums or something to verify that the copy is exact? Thanks again for you help.

It is for remote copying, but you can use it locally just as fine as well. And as for checksums, I am not sure specifically what goes on internally with rsync, but the whole point of rsync is to mirror directories of files. So yeah, it will find the files that are different, and when your done your two directories will be identical.

Here is an rsync tutorial that may be easier to digest than the man page, and it also has an example of using rsync to copy/sync files locally:
http://www.devshed.com/c/a/Administr...on-With-Rsync/

Thanks again for your help! I use rsync for doing backups, but it never thought to use it for doing mirror copying my data.

Just to help anyone else that might need something like this, I did in two steps to verify the data that I copied from one NAS to another was correct. Seems to work pretty well, so I thought I'd share it.

The output looked something like:

I then edited the sums.txt file and switched my NAS1 in the file to NAS2 and ran the 'md5sum -c /tmp/sums.txt' to give me a list of good and bad files.

Hope this help someone...

If you want to run this automatically, try replacing the editing of the MD5 checksum file (sums.txt)
with:

After that, you can put everything in one script and, for instance, make it run once per day or once per week via cron, perhaps even mailing you the results.

Nice addon Timmeke... Now we are down to one step...

What step is that?
Setting up the cron job?

That's easy:
-add all commands in a plain text file, using your favorite text editor
-put your shell as interpreter at the top, using the "shebang" syntax:
ie #!/bin/bash
if you're using bash. (If you don't know your shell, try "echo $SHELL").
-Then type:
crontab -e
This will make you edit your crontab file (contains the cron jobs you want to run).
-To make it run each day, on a fixed time, add the following line:
min hour * * * /path/to/script
Replace "min" by the minute(s) and "hour" by the hour(s) when you want to run the script.
Ranges are allowed (ie 1-5), multiple entries are separated by commas (ie hour 1,4,6,10-14,20).
-To make it run once per week, add the following line instead:
min hour * * day_of_week /path/to/script
day_of_week consists of one or more values between 0 and 7 where 0 is Sunday. Names for weekdays are allowed to.
-For other cron job timing, please look at
man 1 crontab
man 5 crontab
man cron
-Don't forget to save the changes you've made to your crontab file (save your changes when you exit the editor).

Note that, for the "crontab -e" command to work, your user must be allowed to specify cron jobs. This depends on the /etc/cron.allow and /etc/cron.deny files and their contents. Please consult the man pages mentioned above for details.

is '-b' option for md5sum necessary? Linux does not distinguish binary file and text file.

Nowadays, NAS machines based on Windows exist too (unfortunately). So you can never be too careful.

In any case, I'd advise you to use the -b option for binary files.