LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-28-2009, 09:26 AM   #1
proxmity
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Rep: Reputation: 0
Help: Create a shell script to move only files wich has stopped growing


An external service I dont manage pushes mediafiles into a shared directory on my server. I need to move these files into their correct directories automatically. The problem is that if I run my script as a cronjob once every 3 minutes, I notice that the script copies files which are still on their way into my server. So I need to figure out how to have the script check that the files are complete (done downloading) before the script moves the files.

This is what I got so far:
Code:
#!/bin/sh
# Script by proximity 280709.
# Locate correct directory
cd /Volumes/MediaBase/Streambase
# Copy files based on filename.
/bin/mv /Volumes/MediaBase/streamtransit/*DH_*.wmv /Volumes/MediaBase/MediaDB/DH/ > /dev/null 2>&1
/bin/mv /Volumes/MediaBase/streamtransit/*LX_*.wmv /Volumes/MediaBase/MediaDB/LX/ > /dev/null 2>&1
/bin/mv /Volumes/MediaBase/streamtransit/*NA_*.wmv /Volumes/MediaBase/MediaDB/Na/ > /dev/null 2>&1
/bin/mv /Volumes/MediaBase/streamtransit/*NO_*.wmv /Volumes/MediaBase/MediaDB/NO/ > /dev/null 2>&1
/bin/mv /Volumes/MediaBase/streamtransit/*SF_*.wmv /Volumes/MediaBase/MediaDB/SF/ > /dev/null 2>&1
/bin/mv /Volumes/MediaBase/streamtransit/*TD_*.wmv /Volumes/MediaBase/MediaDB/TD/ > /dev/null 2>&1
Any help would be greatly appreciated.
 
Old 07-28-2009, 09:41 AM   #2
PMP
Member
 
Registered: Apr 2009
Location: ~
Distribution: RHEL, Fedora
Posts: 381

Rep: Reputation: 58
Check for the timestamp of the file, if it less than some number say 5 minutes move them
 
Old 07-28-2009, 09:59 AM   #3
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Hello proxmity
Quote:
Originally Posted by PMP View Post
Check for the timestamp of the file, if it less than some number say 5 minutes move them
Try it but it may not work until the file is closed and that will depend on how it is being written. The only robust way is if the program writing the file writes two files, one to indicate completion. For example myfile.part which is receiving the data and myfile; when the upload is finished the program writing the file moves myfile.part to myfile and completion is indicated by myfile.part disappearing.

If the program writing the file doesn't do something like that you could note the file size and, if it hasn't changed after, say, 5 minutes consider it finished. Not robust but maybe "good enough".

Best

Charles
 
Old 07-28-2009, 10:19 AM   #4
onebuck
Moderator
 
Registered: Jan 2005
Location: Midwest USA, Central Illinois
Distribution: SlackwareŽ
Posts: 12,542
Blog Entries: 23

Rep: Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943Reputation: 1943
Hi,

What about using 'sync' then test the timestamp?
 
Old 07-28-2009, 10:42 AM   #5
proxmity
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Original Poster
Rep: Reputation: 0
KInd of you to reply me, thanks.

Quote:
Originally Posted by catkin View Post
Hello proxmity Try it but it may not work until the file is closed and that will depend on how it is being written. The only robust way is if the program writing the file writes two files, one to indicate completion. For example myfile.part which is receiving the data and myfile; when the upload is finished the program writing the file moves myfile.part to myfile and completion is indicated by myfile.part disappearing.

If the program writing the file doesn't do something like that you could note the file size and, if it hasn't changed after, say, 5 minutes consider it finished. Not robust but maybe "good enough".

Best

Charles
Unfortunately, I don't control the service which uploads the files into my server, and that's why I can make the writing program act in any other way than it does today :-/

So I think your second idea is a great one! - which I figure would work out good, except I don't know how to write a script like this. Something I'm looking into now.


Quote:
Originally Posted by onebuck View Post
Hi,

What about using 'sync' then test the timestamp?

Using timestamps is dangerous because the filesize can in one case be only 5 MB, but in the next it might be above 150 GB. So the loading time from the external server may vary to much for a timestamp to be able to be useful, and at the same time provide me with a way to move the files almost as soon as its done loading.

If anyone else have any ideas, know scripts etc, please give me a shout!
I'll make this promise: When I have the solution, I'm not leaving this thread empty and unanswered!
 
Old 07-28-2009, 11:25 AM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Quote:
Originally Posted by onebuck View Post
Hi,

What about using 'sync' then test the timestamp?
Does that update the modification times? Sorry -- I'm not in a position to test it without a lot of messing about.
 
Old 07-28-2009, 11:40 AM   #7
tredegar
LQ 5k Club
 
Registered: May 2003
Location: London, UK
Distribution: Debian "Jessie"
Posts: 6,085

Rep: Reputation: 398Reputation: 398Reputation: 398Reputation: 398
Does the fuser command have anything to offer here?

NAME
fuser − identify processes using files or sockets

man fuser
 
Old 07-28-2009, 11:45 AM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Quote:
Originally Posted by proxmity View Post
So I think your second idea is a great one! - which I figure would work out good, except I don't know how to write a script like this. Something I'm looking into now.
Let us know how it goes; you are going to have fun scripting to maintain a log of file names and time stamps, deleting records applying to moved files. It may prove useful to write those deleted lines out to another log, a log of files moved.

Do you know what the remote systems are using to write the files on your system? I'm wondering if the files are held open while they're being written. Mmm ... that would be too flakey over a network link, unless a local process was doing the holding open and being fed data from a remote process.

A gotcha or two to be wary of ...

A smart file upload system would reserve space on the target system (to save running out of space part way through) so files would never change size!

If you are moving the file to another file system it could take a while, making a long time window for an interrupted transfer to restart with unwanted effects of having the file open for reading and writing. Safer to mv the file to a new name before "moving" it to another file system (= copying and deleting it).

Best

Charles
 
Old 07-28-2009, 01:40 PM   #9
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,053

Rep: Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881
Quote:
Originally Posted by PMP View Post
Check for the timestamp of the file, if it less than some number say 5 minutes move them
Please correct me if I'm wrong, but did you mean, 'if the file timestamp (modified time, presumably) is more than five minutes prior to the current time'?

This idea can only work if you can do the comparison within the time-out time of getting mtime (so, it probably would be a bad idea to get all of the mtimes, compare, copy the file if not within the time out, compare an mtime... as, by the time you get to the last compare, the list could be quite old).

I think the most reliable way to do this would involve lsof. I had though that there was something like a 'remote' option, but my memory must have gone (must get an upgrade); I think with some combination of fd w, +d and -c you can get a list of 'don't move' files; the only exception that comes to mind is if a transfer starts between getting the don't transfer list and getting the list of candidate files. Ideally, you would like prevent new transfers commencing in this time window, which should be short.

(Actually, what you want is probably the opposite of the -X option (-X being thoroughly non-portable, btw), but I'm not sure that is available.

Just be aware that you really, really, will need to read the man page, as and'ing and or'ing the options in lsof doesn't work in the way that you would think.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to move old files, create symbolic link d3coy Linux - Newbie 8 11-09-2011 02:26 PM
shell script to move files from one system to another sytem with file names in a txt coral_km Linux - Newbie 3 02-13-2008 11:23 PM
How to create/delete temp/backup files through a shell script ? Sid2007 Programming 4 10-17-2007 02:55 PM
Very simple shell script - move files to different directories beley Programming 7 11-02-2006 06:24 AM
Need to create a script to move files from one directory to another mustang05 Solaris / OpenSolaris 4 06-16-2006 11:25 AM


All times are GMT -5. The time now is 08:17 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration