Verifying data of storage

floss · 11-10-2009, 04:30 AM

I am moving the data from netapp storage to sun 7410 storage.
I am using rsync to move all the data from old storage to the new storage.
The total size of the data is 4TB.
After i have all the data on the new storage i would like to do an integrity check of the entire data.

What could be the best solution ? I know i could do a md5sum recursively on the entire 4TB data but i guess it would takes ages to complete.

Any suggestion would be highly appreciated.

Thanks,
FLOSS

chiragrk · 11-10-2009, 07:34 AM

The only other thing that comes to my mind if compare the file sizes. Doing md5sum sequentially would take ages on a slow machine, but on a machine with faster CPU it shouldn't be very slow. How about trying to do it parallely? In other words dont wait for md5sum output for file1 before proceeding to check md5sum for file2.

floss · 11-10-2009, 08:44 AM

Thanks for the reply.

I think i would tried that as the storage is accessible from different linux box.

chrism01 · 11-10-2009, 07:34 PM

You may this relevant

Quote:

Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer lqDoes this file need to be updated?rq check.

Sounds like rsync does the check for you.
http://linux.die.net/man/1/rsync

floss · 11-15-2009, 01:43 PM

My management was not convinced about md5sum and rsync. They insisted that i write a script to compare the old and the new storage after rsync.

I wrote a script which look like below. The csv file was later imported to database and compared.

#!/bin/sh

echo `date`
RWXSIZEFILE=/home/floss/rwxsizefile.csv
rm -f $RWXSIZEFILE
touch $RWXSIZEFILE

find /u02 -name "*" -exec stat -c "%A, %s, %n" {} \; | tee -a $RWXSIZEFILE

echo `date`