Short version: How do I ensure that, when I have issued a shell command to copy file F from directory A to mount point C, that F has reached B before proceeding?
(I am thinking sync or fsync, but fsync is a C function, the man page of sync is useless, and the info page of sync does not explain what happens when you flush a mount of a remote directory)
Originally Posted by anomie
Might as well be sure about the file's integrity and sleep well at night.
I would expect integrity checks to be occurring more widely, and tools for this to be plentiful, given how paranoid linux-admins are.
Originally Posted by anomie
[...]it's trivial enough to do a sha1sum(1) of the file before and after the copy, a la:
- generate crypto digest for /path/A/audio_file01
- copy (rather than move) /path/A/audio_file01 -> /path/B
- generate crypto digest for /path/B/audio_file01
- do digests match? make noise and exit if not
- remove /path/A/audio_file01
Some updates on the issue: I did basically what you described. Except, from what I see, when a shell command performing a file transfer to C is complete,
- the file might be "on its way", stored in buffers locally, and
- errors might be introduced in the file during transit / due to ruined server HDD.
I "solved" these issues by, when the transfer to C is complete,
- SCP-ing the file from the remote machine back to local machine
- perform sha1sum on the copy, compare sha sums locally.
No, I am not proud of this solution. I chose this approach after realizing that the remote server does not have the sha1sum command installed. Another reason why this solution is awful is that only one user (named "admin") has privileges to SCP, so for this to work, the local machine needs a) the "admin" password, or b) to have its passwordless public key in the authorized_keys file for "admin" on the the remote machine.
I realized that I did not need cryptographically-strong checksums, so md5sum should be fine. The remote machine has md5sum installed, so an option which does not require the file to be transferred back to local involves having remote do the md5sum on the received file, and make the result available to local.
A much more elegant solution, however, would be to ensure that the file has reached remote when copied to C. One way of doing this would be to remount (unmount, then mount) C, as this flushes buffers. It would be better, however, if I could issue a "flush C" command. However, all I have found in this avenue are
- fsync, which is a C function, not a conveniently-available bash command,
- sync, which flushes all buffers, but is a bit vague about what it does to mounts of remote directories.
If I can ensure that the file has reached the remote machine by flushing the mount point, then I can do the checksum on the file in C locally. However, depending on how md5sum works, this might copy the remote file back to local (which is getting silly again).
Originally Posted by chrism01
There's always rsync; I believe that is very careful about checking whether it went ok.
I envision two uses:
- copy to mount point C & integrity-check,
- copy to remote machine using smb & integrity-check.
For 1), does rsync flush file system buffers upon completion? For 2), how do you do this?