LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-26-2013, 01:44 PM   #1
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Rep: Reputation: 32
Need help fixing this script...


Code:
#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
   if [ ! -f $DEST/"${file}" ] ; then
     {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
   else
       rm -f -- "$SRC/${file}"
   fi
 done

for i in $(ls -l | grep -v "*.uz");
   do
      rm -f -- "$i"
   done
What this script does is take a file from a directory and move it to another directory, if the file exists in the other directory then it deletes the file, no over writing. This works perfectly. I have it setup on a cron job to run every 5 minutes. However, I have discovered an issue and I'm not sure how to correct it.

If you upload a file that takes longer than 5 minutes or is still uploading when that script is activated, then it moves the incomplete file to the destination directory. Thus it's now a corrupt file with the same file name, so even if you do complete the upload, that new completed file in the source directory will now be deleted because the incomplete portion is already at the destination directory.

How can I fix this? I have no idea how to approach the issue.
 
Old 01-26-2013, 02:02 PM   #2
Thor_2.0
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,203
Blog Entries: 23

Rep: Reputation: 279Reputation: 279Reputation: 279
you could grep into the list of processes:
Quote:
ps -A | grep -e [procname]
and see if it's still in there, if not, take further action, otherwise do nothing...
 
Old 01-26-2013, 02:11 PM   #3
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,752

Rep: Reputation: 462Reputation: 462Reputation: 462Reputation: 462Reputation: 462
Perhaps lsof could be of help.
 
Old 01-26-2013, 02:49 PM   #4
lyle_s
Member
 
Registered: Jul 2003
Distribution: Slackware
Posts: 388

Rep: Reputation: 52
You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes.

Lyle.
 
Old 01-26-2013, 04:28 PM   #5
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by Thor_2.0 View Post
you could grep into the list of processes:


and see if it's still in there, if not, take further action, otherwise do nothing...
What procname would I need to look for?

Quote:
Originally Posted by gnashley View Post
Perhaps lsof could be of help.
Going to research this now.


Quote:
Originally Posted by lyle_s View Post
You could use inotifywait to make it more of an event-driven thing instead of having the script run every five minutes.

Lyle.
Going to research this as well.
 
Old 01-26-2013, 04:34 PM   #6
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,159
Blog Entries: 4

Rep: Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760
Monitor file system activity with inotify
 
Old 01-26-2013, 04:45 PM   #7
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by Habitual View Post
Your link is broken.

I found out the syntax I need is something along the lines of

lsof -u [uid] | grep [filename] if exists, do nothing; else move/delete file

Seems to be an easy solution, however I need to figure out how to implement it into my script.
 
Old 01-26-2013, 05:16 PM   #8
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Original Poster
Rep: Reputation: 32
Am I on the right path with this at least?

Code:
#!/bin/bash

 SRC=/home/source/
 DEST=/home2/dest/
 set -e
 cd $SRC || exit 1

find -iname "*.uz" | while read file
 do
   if [ lsof -u 500 | grep "${file}" ] ; then
   elif [ ! -f $DEST/"${file}" ] ; then
     {
        mv -- "${file}" "$DEST/${file}"
        chown ${user}:${group} "$DEST/${file}"
      }
   else
       rm -f -- "$SRC/${file}"
   fi
 done

for i in $(ls -l | grep -v "*.uz");
   do
      rm -f -- "$i"
   done
What I want this to do is this,

Get file's with extensions of .uz, if lsof lists the file name, do nothing, else if, check if file is in destination, then move file to dest, else remove file.

Last edited by Skillz; 01-26-2013 at 05:22 PM.
 
Old 01-28-2013, 06:27 PM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 2,118

Rep: Reputation: 533Reputation: 533Reputation: 533Reputation: 533Reputation: 533Reputation: 533
It would be simpler to check the date on the file - if source is newer than the destination, copy it.
 
Old 01-28-2013, 07:19 PM   #10
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,159
Blog Entries: 4

Rep: Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760
Quote:
Originally Posted by Skillz View Post
Your link is broken.
Yes, sorry, clipboard confusion...

http://www.ibm.com/developerworks/li...ify/index.html is correct.
 
Old 01-28-2013, 11:16 PM   #11
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by jpollard View Post
It would be simpler to check the date on the file - if source is newer than the destination, copy it.
That's not the issue at hand. The issue is determining if the file has completed transferring before moving it to the destination. Also the destination files are considered "originals" and should not be over written. Otherwise, I would just allow uploads directly to the destination and prevent deleting of those files.

Basically these files are game files for a game server, you download the files off the web server instead of the game client so the downloads go much faster and don't effect the bandwidth of the game server assuming the web server is on different hardware and a different connection.

I'm concerned with people with bad intentions just deleting the files off the web server (its happened before) or uploading bad/malicious files just to be a tool. Therefore I only allow them to upload files that are not already on the redirect, since most times those files are something that they've created for the game so it's unique. I do allow them to submit files for deletion with a reason on why it should be deleted.

Basically, the method I have now works except if the script tries to transfer the file before it's done uploading. Which is what I am trying to use LSOF for. To see if the file is still being accessed/uploaded and skip it to the next one until it's done uploading.

Last edited by Skillz; 01-28-2013 at 11:20 PM.
 
Old 01-29-2013, 05:26 AM   #12
Thor_2.0
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,203
Blog Entries: 23

Rep: Reputation: 279Reputation: 279Reputation: 279
Quote:
What procname would I need to look for?
The name of your script, what it was started up with...
If you'd start Blender from the console (or the menu) you'd see "Blender" inthe list...
 
Old 01-29-2013, 02:28 PM   #13
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,159
Blog Entries: 4

Rep: Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760
Code:
sudo lsof +D /path/to/directory
will/should show you what's writing, or using the /path/to/directory.

If there's a "hit", it's being "used".

HTH.
 
1 members found this post helpful.
Old 01-30-2013, 04:28 PM   #14
Skillz
Member
 
Registered: Sep 2007
Posts: 234

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by Habitual View Post
Code:
sudo lsof +D /path/to/directory
will/should show you what's writing, or using the /path/to/directory.

If there's a "hit", it's being "used".

HTH.
So this would prevent moving of files anytime the directory is being written to, correct? Which means if uploading is busy, it may take a while to transfer files?
 
Old 01-31-2013, 12:13 PM   #15
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,159
Blog Entries: 4

Rep: Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760Reputation: 760
Quote:
Originally Posted by Skillz View Post
So this would prevent moving of files anytime the directory is being written to, correct?
Absolutely incorrect.
A real example:
Code:
lsof +D /tmp
COMMAND   PID   USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
java     3010 zimbra mem    REG  202,1    32768 147517 /tmp/hsperfdata_root/3010
java    20143 zimbra mem    REG  202,1    32768 132734 /tmp/hsperfdata_zimbra/20143
shows that java is writing, or has open these 2 files in /tmp:
Code:
/tmp/hsperfdata_root/3010
/tmp/hsperfdata_zimbra/20143
This method negates
Quote:
Originally Posted by Skillz View Post
Get file's with extensions of .uz
if lsof lists the file name, do nothing
I'd use it like so:
Code:
lsof +D "$SRC" | grep "\.uz"
lsof +D "$DEST" | grep "\.uz"
So to summarize: I think you should re-think your pseudo-code.
Code:
Get file's with extensions of .uz
if lsof lists the file name do nothing
else check if file is in destination else
move file to dest
else remove file.
b/c "lsof lists the file name do nothing" is likely to be unreliable. the file could be there AND be written to at the same time. From the first One to the last Zero...
so my suggestion is check to see if anything is open and is writing to a .uz extension in "$DEST" using
lsof +D "$SRC" | grep "\.uz" before any further logic is applied or executed.

I'm certain that I hacked the crap out of this or just didn't help you at all. All I want to make clear is that there are as many ways to do this as there are Admins.

Some one smarter or more 'efficient' at it than myself will come along, and I hope we both get an "aha!, or of course!" moment out of it.

Have a Great Day and peek at http://tldp.org/LDP/abs/html/ioredirintro.html

John

Last edited by Habitual; 02-02-2013 at 01:17 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] fixing ranges using shell script kswapnadevi Linux - Newbie 3 12-03-2010 01:24 PM
need help with fixing php script frieza Programming 3 01-27-2008 06:15 PM
Script for fixing broken character encodings in emails pschoenb Linux - Server 1 02-28-2007 04:37 PM
need help fixing and optimizing AWK script hedpe Programming 5 08-29-2006 08:30 AM
can somebody help me in fixing the script jdara1 Linux - General 2 10-23-2003 12:18 PM


All times are GMT -5. The time now is 01:56 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration