[SOLVED] Verifying physical integrity of backed up files

SlowCoder · 06-18-2021, 12:15 PM

I bought a new USB drive for primary backups. I used 'rsync -av src dest', and all appears to have copied over.

I want to verify the backup files are actually stored correctly (no on-disk errors), and are exactly the same as the source.

Searches bring up rsync -c or --checksum. But diving into it, this function only works on the sending and receiving processes, not the files after they have been written to disk.

How can I verify the checksums of all my backup files against my source files, post write to disk?

business_kid · 06-18-2021, 02:23 PM

Code:

rsync -avc src dest

where src must be the canonical file version. Unless your media is dodgy or you saw errors, your files are good. But go ahead if you want to check.

smallpond · 06-18-2021, 03:04 PM

Put the following script named diff12.sh in a directory in your path:

Code:

#!/bin/bash
src=$1
dest=$DESTPATH${src#$SRCPATH}
cmp $src $dest

then set the directories of the source and dest and compare each file in src to see if dest matches.

Code:

export SRCPATH=src 
export DESTPATH=dest 
find $SRCPATH -type f diff12.sh {} \;

rnturn · 06-18-2021, 07:03 PM

Quote:

Originally Posted by SlowCoder

How can I verify the checksums of all my backup files against my source files, post write to disk?

A cheap-n-dirty scriptable post-backup process could be used. You'd need to make a list of the files on the backup drive and obtain their checksums (md5sum or better), sort the result and save the results in a file ("backup.cksums"). Do the same for the source device/filesystem/tree ("source.cksums"). A check for files that are different in the two collections of files would be to issue:

Code:

$ cat *.cksums | awk '{print $1}' | sort | uniq -c > freq.lis
$ grep ' 1 ' freq.lis

You want the vast majority of the checksums that result from the above steps to be found in pairs (or more which means you have identical copies sitting in different subdirectories). Ideally, there are the same number of files in the source and the backup "trees" and the checksums are identical across the trees so everything in "freq.lis" will be prefixed with "2"---perfect backup! If anything results from the "grep ' 1 '", issue "grep some-checksum" against each of the ".cksums' files to find out what files differed.

HTH...

syg00 · 06-19-2021, 02:08 AM

Quote:

Originally Posted by SlowCoder

Searches bring up rsync -c or --checksum. But diving into it, this function only works on the sending and receiving processes, not the files after they have been written to disk.

I take it your delving didn't include the manpage.

Quote:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving
side by checking a whole-file checksum that is generated as the file is transferred, but that automatic
after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.

SlowCoder · 06-20-2021, 12:39 PM

Quote:

Originally Posted by business_kid

Code:

rsync -avc src dest

where src must be the canonical file version. Unless your media is dodgy or you saw errors, your files are good. But go ahead if you want to check.

Quote:

Originally Posted by syg00

I take it your delving didn't include the manpage.

Yes, I did.

To answer both quotes: Apparently this is only an ack with checksum between the sender and receiver protocols, and does not verify the physical write to hardware (hdd, ssd, etc.)

rtmistler · 06-20-2021, 02:28 PM

Quote:

Originally Posted by SlowCoder

Apparently this is only an ack with checksum between the sender and receiver protocols, and does not verify the physical write to hardware (hdd, ssd, etc.)

Where are you getting this information?

astrogeek · 06-20-2021, 02:39 PM

Quote:

Originally Posted by SlowCoder

Apparently this is only an ack with checksum between the sender and receiver protocols, and does not verify the physical write to hardware (hdd, ssd, etc.)

Assuming that to be true and to mean what you think (I don't know), then the simplest way that comes immediately to mind would be to simply run your rsync command a second time after the initial backup and see if it finds anything different.

business_kid · 06-21-2021, 06:56 AM

@OP: If you use the -c option and run rsync a second time like I told you back in post #2(!) calculating the checksum checks the write. Is there a difficulty with that?

SlowCoder · 06-21-2021, 10:42 AM

Quote:

Originally Posted by astrogeek

Assuming that to be true and to mean what you think (I don't know), then the simplest way that comes immediately to mind would be to simply run your rsync command a second time after the initial backup and see if it finds anything different.

Quote:

Originally Posted by business_kid

@OP: If you use the -c option and run rsync a second time like I told you back in post #2(!) calculating the checksum checks the write. Is there a difficulty with that?

Now, this does make sense! I would just need to 'sync' the cached data in between to ensure the data has been written.