LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   Need help fixing this script... (http://www.linuxquestions.org/questions/programming-9/need-help-fixing-this-script-4175447351/)

Skillz 01-26-2013 01:44 PM

Need help fixing this script...
 
Code:

#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
  if [ ! -f $DEST/"${file}" ] ; then
    {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
  else
      rm -f -- "$SRC/${file}"
  fi
 done

for i in $(ls -l | grep -v "*.uz");
  do
      rm -f -- "$i"
  done

What this script does is take a file from a directory and move it to another directory, if the file exists in the other directory then it deletes the file, no over writing. This works perfectly. I have it setup on a cron job to run every 5 minutes. However, I have discovered an issue and I'm not sure how to correct it.

If you upload a file that takes longer than 5 minutes or is still uploading when that script is activated, then it moves the incomplete file to the destination directory. Thus it's now a corrupt file with the same file name, so even if you do complete the upload, that new completed file in the source directory will now be deleted because the incomplete portion is already at the destination directory.

How can I fix this? I have no idea how to approach the issue.

Thor_2.0 01-26-2013 02:02 PM

you could grep into the list of processes:
Quote:

ps -A | grep -e [procname]
and see if it's still in there, if not, take further action, otherwise do nothing...

gnashley 01-26-2013 02:11 PM

Perhaps lsof could be of help.

lyle_s 01-26-2013 02:49 PM

You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes.

Lyle.

Skillz 01-26-2013 04:28 PM

Quote:

Originally Posted by Thor_2.0 (Post 4878111)
you could grep into the list of processes:


and see if it's still in there, if not, take further action, otherwise do nothing...

What procname would I need to look for?

Quote:

Originally Posted by gnashley (Post 4878117)
Perhaps lsof could be of help.

Going to research this now.


Quote:

Originally Posted by lyle_s (Post 4878138)
You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes.

Lyle.

Going to research this as well.

Habitual 01-26-2013 04:34 PM

Monitor file system activity with inotify

Skillz 01-26-2013 04:45 PM

Quote:

Originally Posted by Habitual (Post 4878200)

Your link is broken.

I found out the syntax I need is something along the lines of

lsof -u [uid] | grep [filename] if exists, do nothing; else move/delete file

Seems to be an easy solution, however I need to figure out how to implement it into my script.

Skillz 01-26-2013 05:16 PM

Am I on the right path with this at least?

Code:

#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
  if [ lsof -u 500 | grep "${file}" ] ; then
  elif [ ! -f $DEST/"${file}" ] ; then
    {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
  else
      rm -f -- "$SRC/${file}"
  fi
 done

for i in $(ls -l | grep -v "*.uz");
  do
      rm -f -- "$i"
  done

What I want this to do is this,

Get file's with extensions of .uz, if lsof lists the file name, do nothing, else if, check if file is in destination, then move file to dest, else remove file.

jpollard 01-28-2013 06:27 PM

It would be simpler to check the date on the file - if source is newer than the destination, copy it.

Habitual 01-28-2013 07:19 PM

Quote:

Originally Posted by Skillz (Post 4878202)
Your link is broken.

Yes, sorry, clipboard confusion...

http://www.ibm.com/developerworks/li...ify/index.html is correct.

Skillz 01-28-2013 11:16 PM

Quote:

Originally Posted by jpollard (Post 4879404)
It would be simpler to check the date on the file - if source is newer than the destination, copy it.

That's not the issue at hand. The issue is determining if the file has completed transferring before moving it to the destination. Also the destination files are considered "originals" and should not be over written. Otherwise, I would just allow uploads directly to the destination and prevent deleting of those files.

Basically these files are game files for a game server, you download the files off the web server instead of the game client so the downloads go much faster and don't effect the bandwidth of the game server assuming the web server is on different hardware and a different connection.

I'm concerned with people with bad intentions just deleting the files off the web server (its happened before) or uploading bad/malicious files just to be a tool. Therefore I only allow them to upload files that are not already on the redirect, since most times those files are something that they've created for the game so it's unique. I do allow them to submit files for deletion with a reason on why it should be deleted.

Basically, the method I have now works except if the script tries to transfer the file before it's done uploading. Which is what I am trying to use LSOF for. To see if the file is still being accessed/uploaded and skip it to the next one until it's done uploading.

Thor_2.0 01-29-2013 05:26 AM

Quote:

What procname would I need to look for?
The name of your script, what it was started up with...
If you'd start Blender from the console (or the menu) you'd see "Blender" inthe list...

Habitual 01-29-2013 02:28 PM

Code:

sudo lsof +D /path/to/directory
will/should show you what's writing, or using the /path/to/directory.

If there's a "hit", it's being "used".

HTH.

Skillz 01-30-2013 04:28 PM

Quote:

Originally Posted by Habitual (Post 4880103)
Code:

sudo lsof +D /path/to/directory
will/should show you what's writing, or using the /path/to/directory.

If there's a "hit", it's being "used".

HTH.

So this would prevent moving of files anytime the directory is being written to, correct? Which means if uploading is busy, it may take a while to transfer files?

Habitual 01-31-2013 12:13 PM

Quote:

Originally Posted by Skillz (Post 4880931)
So this would prevent moving of files anytime the directory is being written to, correct?

Absolutely incorrect.
A real example:
Code:

lsof +D /tmp
COMMAND  PID  USER  FD  TYPE DEVICE SIZE/OFF  NODE NAME
java    3010 zimbra mem    REG  202,1    32768 147517 /tmp/hsperfdata_root/3010
java    20143 zimbra mem    REG  202,1    32768 132734 /tmp/hsperfdata_zimbra/20143

shows that java is writing, or has open these 2 files in /tmp:
Code:

/tmp/hsperfdata_root/3010
/tmp/hsperfdata_zimbra/20143

This method negates
Quote:

Originally Posted by Skillz (Post 4878207)
Get file's with extensions of .uz
if lsof lists the file name, do nothing

I'd use it like so:
Code:

lsof +D "$SRC" | grep "\.uz"
lsof +D "$DEST" | grep "\.uz"

So to summarize: I think you should re-think your pseudo-code.
Code:

Get file's with extensions of .uz
if lsof lists the file name do nothing
else check if file is in destination else
move file to dest
else remove file.

b/c "lsof lists the file name do nothing" is likely to be unreliable. the file could be there AND be written to at the same time. From the first One to the last Zero...
so my suggestion is check to see if anything is open and is writing to a .uz extension in "$DEST" using
lsof +D "$SRC" | grep "\.uz" before any further logic is applied or executed.

I'm certain that I hacked the crap out of this or just didn't help you at all. All I want to make clear is that there are as many ways to do this as there are Admins.

Some one smarter or more 'efficient' at it than myself will come along, and I hope we both get an "aha!, or of course!" moment out of it.

Have a Great Day and peek at http://tldp.org/LDP/abs/html/ioredirintro.html

John


All times are GMT -5. The time now is 07:17 AM.