LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 04-30-2005, 06:29 PM   #1
babag
Member
 
Registered: Aug 2003
Posts: 365

Rep: Reputation: 30
verify before continue - how to?


been working with large numbers of files - moving, copying, sorting, renaming.
start with this command to grab from my xp box and move into the linux box
for processing:

ls | xargs -i -t cp {} /home/babag/BASHing/master_frames

have found that it misses some files. there does not appear to be a pattern to
the files that are missed. files are numbered video frames with the naming
protocol:

Buddies_0123.bmp

the 0123 represents the numbering of the files. the next frame would be 0124,
the next 0125, etc.

must be some kind of network glitch, possibly speed related? my question is,
what commands would i use to verify that each file is indeed copied before the
command moves on to the next file? i'd like to keep retrying missed files until
they're copied before moving on to the next file. should i build this into a script
rather than performing it as a single command line command?

and in a related question, here's a short script i'd also like to build verification
into:

Code:
 
#!/bin/bash

# This script removes the duplicate .bmp frames from the film 
# transfer 3:2 pulldown sequence.

cd /home/babag/BASHing/master_frames

for i in *.bmp ; do
   j=`echo "$i" | awk -F"." '{print$1}'`
   k=`echo "$j" | awk -F"_" '{printf("%d", $2)}'`

   if [ $((k % 5)) == 0 ]; then
      rm -v $i
   fi

done
thanks,
BabaG

Last edited by babag; 04-30-2005 at 06:32 PM.
 
Old 05-01-2005, 06:19 AM   #2
wapcaplet
Guru
 
Registered: Feb 2003
Location: Colorado Springs, CO
Distribution: Gentoo
Posts: 2,018

Rep: Reputation: 48
You could use a loop in bash, something like this:

Code:
for FILE in $( ls ); do
  # Keep trying until destination file exists
  while [[ ! -e /destination/directory/$FILE ]]; do
    cp $FILE /destination/directory/$FILE
  done
done
This code isn't very robust - you'd want to do some other sanity checks also, to make sure you have read/write permission to the appropriate places, or that there's enough space to copy. This code would loop forever if there's a problem writing to the destination. If you have any filenames with spaces, you'll probably want double-quotes around $FILE wherever it appears.

You could also compare the size of the source/destination files. If you really want to make sure it copied correctly, you could run an md5 sum on each one and compare those.

Hope this helps!
 
Old 05-01-2005, 08:09 AM   #3
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
There is a program called seq that you could use to produce a sequence of numbers for a for loop.

for num in $(seq -w 900 1200); do
if [ ! -f buddie_0${num} ] ; then
cp $FILE /destination/directory/$FILE
fi; done

Like you said, you may want to add other tests, such as permissions.

Or you could use the find command instead of ls to produce a list which match more than one criteria such as ownership.


find <source-dir> -iname "buddies_[[:digit:]]*" -maxdepth 1 -user wapcaplet -print0 | xargs --null -r cp --reply=query "{}" <destination-dir>

If you have a very large number of files, you may want to use the xargs option --max-args=max-args.
If there aren't to many files you don't need xargs.

find <source-dir> -iname "buddies_[[:digit:]]*" -maxdepth 1 -user wapcaplet -exec cp "{}" <destination-dir> \;
 
Old 05-01-2005, 06:53 PM   #4
babag
Member
 
Registered: Aug 2003
Posts: 365

Original Poster
Rep: Reputation: 30
thanks wapcaplet and jschiwal. tried wapcaplet's little script. while it did
copy files, it still missed some. would you mind pointing out how the
verification is taking place here? trying to learn. also, jschiwal, since i
seem to be having a difficult time getting things to work the way everybody
expects, could you please explain your loop in a bit more detail. i'm a noob
and am obviously having trouble getting pretty basic stuff to work.

my loops continue even when they've missed a file. i'd like the operation
to refuse to continue until it's certain that each numbered file has really
been copied. it would seem that there should be two places to catch this
and i don't know where it's failing. the first is the source directory; maybe
the script is not grabbing the source file as it should. second is the
destination directory; maybe the failure is on this end.

after much frustration, these are the two places i'd like to verify the
process. help in doing so is much appreciated. btw jschiwal, there are
indeed many files. these are individual picture files creates from the
frames of videotape sequences, many thousands. the directory i'm
testing with currently has about 30,000.

thanks again,
BabaG

Last edited by babag; 05-01-2005 at 06:54 PM.
 
Old 05-01-2005, 08:34 PM   #5
babag
Member
 
Registered: Aug 2003
Posts: 365

Original Poster
Rep: Reputation: 30
well, i changed my approach. no matter what i did, i couldn't seem to get my
system to correctly copy all of my files without skipping some for no apparent
reason. since the files are all of the same type and named using a common
naming protocol, i tried assigning the specific name of each file to the copy
command rather than saying generally, "copy everything one after the other."

if anyone can help clean up this sloppy code i'd appreciate it. frankly, this
seems like a lot more work to go through than should be necessary just to
copy files.

thanks,
BabaG

Code:
#!/bin/bash

# Script to give a filename and copy it from 
# Filename protocol is Buddies_0123.bmp

sourdir=/home/babag/VIDEO/Buddies/BMPSEQ/Part-6/working2
destdir=/home/babag/BASHing/master_frames

cd $sourdir

sframe=1
eframe=29434
prefix=Buddies
extension=bmp

q=1
z=10
zz=100
zzz=1000
zzzz=10000
zzzzz=100000

for num in $(seq $sframe $eframe); do
      
   if [ $num -lt $z ]; then                         # add leading 0's
      p="0000"
   else
      if [ $num -lt $zz ]; then
         p="000"
      else
         if [ $num -lt $zzz ]; then
            p="00"
         else
            if [ $num -lt $zzzz ]; then
               p="0"
            else
	       if [ $num -lt $zzzzz ]; then
                  p=""
               fi	          
	    fi
         fi
      fi
   fi
   
   
   sourfile=$prefix"_"$p$num"."$extension
   cp -v "$sourfile" $destdir                        # copy to mdk

done

Last edited by babag; 05-02-2005 at 12:11 AM.
 
Old 05-01-2005, 10:15 PM   #6
Dave Kelly
Member
 
Registered: Aug 2004
Location: Todd Mission Texas
Distribution: Linspire
Posts: 213

Rep: Reputation: 31
Is there a reason that something like this won't work?

cp /home/babag/VIDEO/Buddies/BMPSEQ/Part-6/working2/* \
/home/babag/BASHing/master_frames
 
Old 05-02-2005, 12:09 AM   #7
babag
Member
 
Registered: Aug 2003
Posts: 365

Original Poster
Rep: Reputation: 30
dave,

the only way to copy reliably that i've found is by giving the command the specific
filenames. i'm beginning to think that what's going on has to do with a network issue
interrupting the process. something like being a passenger in a car and looking out
the window and counting the passing telephone poles. the driver says something
and you turn away to answer. when you turn back you continue counting but you've
missed a pole. only thing i can think of. when i go back and try to manually copy one
of the missed ones it works fine so it's not a permissions issue.

thanks,
BabaG
 
Old 05-02-2005, 10:33 PM   #8
wapcaplet
Guru
 
Registered: Feb 2003
Location: Colorado Springs, CO
Distribution: Gentoo
Posts: 2,018

Rep: Reputation: 48
I know there's a limitation on how long a bash command-line can be; when you use a command like this:

Code:
cp *.bmp /some/directory
bash "expands" the asterisk into a complete list of all the files that match the given name, so the end result is something like this:

Code:
cp 001.bmp 002.bmp 003.bmp 004.bmp ... /some/directory
Obviously, if there are a lot of files, this command gets very long. I don't know if this is the limitation you are encountering, though.

Here's another approach: try putting all the files to be copied in a .tar file, like so:

Code:
tar -cvvf all.tar *.bmp
Then you'll only have one large file (all.tar) to copy.
 
Old 05-05-2005, 05:28 AM   #9
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
# The $( ... ) construct will execute the command inside and provide the stdio output as text to be included in the line.
# In this case, produces a sequence of numbers that get assigned to $num for each iteration through the loop.
# for num in $(seq -w 900 1200); do

# This is a test for the existence of the file. The "-f" tests for a regular file. The "!" character negates the test, so if the file doesn't exist, the statements following "then" are executed. The if/then block is terminated by "fi" which simply is "if" spelt back-wards.
if [ ! -f buddie_0${num} ] ; then

cp $FILE /destination/directory/$FILE

fi; done


-----

If the sequence is too long, that can cause a problem, however it will fail right away. You could add something to the line that does the copying.
cp $FILE /destination/directory/$FILE || echo >>logfile "Copy $FILE failed"

The "||" symbol is for 'or'. If the first part fails, the second part succeeds.
Suppose that you have downloaded a bunch of files on Usenet. They are scenic pictures, but some of the names are too generic so you have files like "picture-1.jpg", "picture-1_copy_1.jpg".
You could have a loop like:
for picture in *_copy_1.jpg; do
cmp "$picture" "${picture/_copy-1/}" && rm "$file"
done

In this case I used the "cmp" command to compare two files. If they are identical, the cmp command will return success and the *_copy_1.jpg file will be deleted. If they had been different, the next statement would be executed instead and the loop would continue.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Continue BLFS 6.1 satimis Linux From Scratch 4 08-17-2005 03:18 AM
Slackware will continue GinEric Slackware 11 12-15-2004 06:28 PM
Will not continue installation neoanderson19 Linux - Software 3 08-07-2004 10:45 AM
Samba problems continue - please help Eradicator Linux - Networking 6 11-04-2003 07:38 PM
Dazed and confused, but trying to continue MegamanNL Debian 1 10-23-2003 03:27 PM


All times are GMT -5. The time now is 06:00 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration