Need help fixing this script...

#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
   if [ ! -f $DEST/"${file}" ] ; then
     {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
   else
       rm -f -- "$SRC/${file}"
   fi
 done

for i in $(ls -l | grep -v "*.uz");
   do
      rm -f -- "$i"
   done

What this script does is take a file from a directory and move it to another directory, if the file exists in the other directory then it deletes the file, no over writing. This works perfectly. I have it setup on a cron job to run every 5 minutes. However, I have discovered an issue and I'm not sure how to correct it.

If you upload a file that takes longer than 5 minutes or is still uploading when that script is activated, then it moves the incomplete file to the destination directory. Thus it's now a corrupt file with the same file name, so even if you do complete the upload, that new completed file in the source directory will now be deleted because the incomplete portion is already at the destination directory.

How can I fix this? I have no idea how to approach the issue.

you could grep into the list of processes:

ps -A | grep -e [procname]

and see if it's still in there, if not, take further action, otherwise do nothing...

Perhaps lsof could be of help.

You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes.

Lyle.

Originally Posted by Thor_2.0 you could grep into the list of processes: and see if it's still in there, if not, take further action, otherwise do nothing...

What procname would I need to look for?

Originally Posted by gnashley Perhaps lsof could be of help.

Going to research this now.

Originally Posted by lyle_s You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes. Lyle.

Going to research this as well.

Monitor file system activity with inotify

Originally Posted by Habitual Monitor file system activity with inotify

Your link is broken.

I found out the syntax I need is something along the lines of

lsof -u [uid] | grep [filename] if exists, do nothing; else move/delete file

Seems to be an easy solution, however I need to figure out how to implement it into my script.

Am I on the right path with this at least?

#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
   if [ lsof -u 500 | grep "${file}" ] ; then
   elif [ ! -f $DEST/"${file}" ] ; then
     {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
   else
       rm -f -- "$SRC/${file}"
   fi
 done

for i in $(ls -l | grep -v "*.uz");
   do
      rm -f -- "$i"
   done

What I want this to do is this,

Get file's with extensions of .uz, if lsof lists the file name, do nothing, else if, check if file is in destination, then move file to dest, else remove file.

It would be simpler to check the date on the file - if source is newer than the destination, copy it.

Originally Posted by Skillz Your link is broken.

Yes, sorry, clipboard confusion...

http://www.ibm.com/developerworks/li...ify/index.html is correct.

Originally Posted by jpollard It would be simpler to check the date on the file - if source is newer than the destination, copy it.

That's not the issue at hand. The issue is determining if the file has completed transferring before moving it to the destination. Also the destination files are considered "originals" and should not be over written. Otherwise, I would just allow uploads directly to the destination and prevent deleting of those files.

Basically these files are game files for a game server, you download the files off the web server instead of the game client so the downloads go much faster and don't effect the bandwidth of the game server assuming the web server is on different hardware and a different connection.

I'm concerned with people with bad intentions just deleting the files off the web server (its happened before) or uploading bad/malicious files just to be a tool. Therefore I only allow them to upload files that are not already on the redirect, since most times those files are something that they've created for the game so it's unique. I do allow them to submit files for deletion with a reason on why it should be deleted.

Basically, the method I have now works except if the script tries to transfer the file before it's done uploading. Which is what I am trying to use LSOF for. To see if the file is still being accessed/uploaded and skip it to the next one until it's done uploading.

What procname would I need to look for?

The name of your script, what it was started up with...
If you'd start Blender from the console (or the menu) you'd see "Blender" inthe list...

sudo lsof +D /path/to/directory

will/should show you what's writing, or using the /path/to/directory.

If there's a "hit", it's being "used".

HTH.

Originally Posted by Habitual sudo lsof +D /path/to/directory will/should show you what's writing, or using the /path/to/directory. If there's a "hit", it's being "used". HTH.

So this would prevent moving of files anytime the directory is being written to, correct? Which means if uploading is busy, it may take a while to transfer files?

Originally Posted by Skillz So this would prevent moving of files anytime the directory is being written to, correct?

Absolutely incorrect.
A real example:

lsof +D /tmp
COMMAND   PID   USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
java     3010 zimbra mem    REG  202,1    32768 147517 /tmp/hsperfdata_root/3010
java    20143 zimbra mem    REG  202,1    32768 132734 /tmp/hsperfdata_zimbra/20143

shows that java is writing, or has open these 2 files in /tmp:

/tmp/hsperfdata_root/3010
/tmp/hsperfdata_zimbra/20143

This method negates

Originally Posted by Skillz Get file's with extensions of .uz if lsof lists the file name, do nothing

I'd use it like so:

lsof +D "$SRC" | grep "\.uz"
lsof +D "$DEST" | grep "\.uz"

So to summarize: I think you should re-think your pseudo-code.

Get file's with extensions of .uz
if lsof lists the file name do nothing
else check if file is in destination else
move file to dest
else remove file.

b/c "lsof lists the file name do nothing" is likely to be unreliable. the file could be there AND be written to at the same time. From the first One to the last Zero...
so my suggestion is check to see if anything is open and is writing to a .uz extension in "$DEST" using
lsof +D "$SRC" | grep "\.uz" before any further logic is applied or executed.

I'm certain that I hacked the crap out of this or just didn't help you at all. All I want to make clear is that there are as many ways to do this as there are Admins.

Some one smarter or more 'efficient' at it than myself will come along, and I hope we both get an "aha!, or of course!" moment out of it.

Have a Great Day and peek at http://tldp.org/LDP/abs/html/ioredirintro.html

John