verify before continue - how to?
 

been working with large numbers of files - moving, copying, sorting, renaming.
start with this command to grab from my xp box and move into the linux box
for processing:

ls | xargs -i -t cp {} /home/babag/BASHing/master_frames

have found that it misses some files. there does not appear to be a pattern to
the files that are missed. files are numbered video frames with the naming
protocol:

Buddies_0123.bmp

the 0123 represents the numbering of the files. the next frame would be 0124,
the next 0125, etc.

must be some kind of network glitch, possibly speed related? my question is,
what commands would i use to verify that each file is indeed copied before the
command moves on to the next file? i'd like to keep retrying missed files until
they're copied before moving on to the next file. should i build this into a script
rather than performing it as a single command line command?

and in a related question, here's a short script i'd also like to build verification
into:

thanks,
BabaG

You could use a loop in bash, something like this:

This code isn't very robust - you'd want to do some other sanity checks also, to make sure you have read/write permission to the appropriate places, or that there's enough space to copy. This code would loop forever if there's a problem writing to the destination. If you have any filenames with spaces, you'll probably want double-quotes around $FILE wherever it appears.

You could also compare the size of the source/destination files. If you really want to make sure it copied correctly, you could run an md5 sum on each one and compare those.

Hope this helps!

There is a program called seq that you could use to produce a sequence of numbers for a for loop.

for num in $(seq -w 900 1200); do
if [ ! -f buddie_0${num} ] ; then
cp $FILE /destination/directory/$FILE
fi; done

Like you said, you may want to add other tests, such as permissions.

Or you could use the find command instead of ls to produce a list which match more than one criteria such as ownership.


find <source-dir> -iname "buddies_[[:digit:]]*" -maxdepth 1 -user wapcaplet -print0 | xargs --null -r cp --reply=query "{}" <destination-dir>

If you have a very large number of files, you may want to use the xargs option --max-args=max-args.
If there aren't to many files you don't need xargs.

find <source-dir> -iname "buddies_[[:digit:]]*" -maxdepth 1 -user wapcaplet -exec cp "{}" <destination-dir> \;

thanks wapcaplet and jschiwal. tried wapcaplet's little script. while it did
copy files, it still missed some. would you mind pointing out how the
verification is taking place here? trying to learn. also, jschiwal, since i
seem to be having a difficult time getting things to work the way everybody
expects, could you please explain your loop in a bit more detail. i'm a noob
and am obviously having trouble getting pretty basic stuff to work.

my loops continue even when they've missed a file. i'd like the operation
to refuse to continue until it's certain that each numbered file has really
been copied. it would seem that there should be two places to catch this
and i don't know where it's failing. the first is the source directory; maybe
the script is not grabbing the source file as it should. second is the
destination directory; maybe the failure is on this end.

after much frustration, these are the two places i'd like to verify the
process. help in doing so is much appreciated. btw jschiwal, there are
indeed many files. these are individual picture files creates from the
frames of videotape sequences, many thousands. the directory i'm
testing with currently has about 30,000.

thanks again,
BabaG

well, i changed my approach. no matter what i did, i couldn't seem to get my
system to correctly copy all of my files without skipping some for no apparent
reason. since the files are all of the same type and named using a common
naming protocol, i tried assigning the specific name of each file to the copy
command rather than saying generally, "copy everything one after the other."

if anyone can help clean up this sloppy code i'd appreciate it. frankly, this
seems like a lot more work to go through than should be necessary just to
copy files.

thanks,
BabaG

Is there a reason that something like this won't work?

cp /home/babag/VIDEO/Buddies/BMPSEQ/Part-6/working2/* \
/home/babag/BASHing/master_frames

dave,

the only way to copy reliably that i've found is by giving the command the specific
filenames. i'm beginning to think that what's going on has to do with a network issue
interrupting the process. something like being a passenger in a car and looking out
the window and counting the passing telephone poles. the driver says something
and you turn away to answer. when you turn back you continue counting but you've
missed a pole. only thing i can think of. when i go back and try to manually copy one
of the missed ones it works fine so it's not a permissions issue.

thanks,
BabaG

I know there's a limitation on how long a bash command-line can be; when you use a command like this:

bash "expands" the asterisk into a complete list of all the files that match the given name, so the end result is something like this:

Obviously, if there are a lot of files, this command gets very long. I don't know if this is the limitation you are encountering, though.

Here's another approach: try putting all the files to be copied in a .tar file, like so:

Then you'll only have one large file (all.tar) to copy.

# The $( ... ) construct will execute the command inside and provide the stdio output as text to be included in the line.
# In this case, produces a sequence of numbers that get assigned to $num for each iteration through the loop.
# for num in $(seq -w 900 1200); do

# This is a test for the existence of the file. The "-f" tests for a regular file. The "!" character negates the test, so if the file doesn't exist, the statements following "then" are executed. The if/then block is terminated by "fi" which simply is "if" spelt back-wards.
if [ ! -f buddie_0${num} ] ; then

cp $FILE /destination/directory/$FILE

fi; done


-----

If the sequence is too long, that can cause a problem, however it will fail right away. You could add something to the line that does the copying.
cp $FILE /destination/directory/$FILE || echo >>logfile "Copy $FILE failed"

The "||" symbol is for 'or'. If the first part fails, the second part succeeds.
Suppose that you have downloaded a bunch of files on Usenet. They are scenic pictures, but some of the names are too generic so you have files like "picture-1.jpg", "picture-1_copy_1.jpg".
You could have a loop like:
for picture in *_copy_1.jpg; do
cmp "$picture" "${picture/_copy-1/}" && rm "$file"
done

In this case I used the "cmp" command to compare two files. If they are identical, the cmp command will return success and the *_copy_1.jpg file will be deleted. If they had been different, the next statement would be executed instead and the loop would continue.