Efficient way of doing this?

tuxtutorials · 04-26-2010, 02:57 PM

Hello all, I would like some input on a small script I wrote to move a file that is received daily to one of our servers. My main question is regarding the file comparison piece where I do a compare between the two variables FILE_DATE and SYSTEM_DATE. I am pretty sure there is an easier way to do this but by brain couldn't figure it out. Any input would be great.

Code:

#!/bin/bash
#Description: Script to parse incoming file and archive if recent.


DATE=`date +%m-%d-%y`
INBOUND_DIR=/home/dropdir
ARCHIVE_DIR=/home/dropdir/archive
LOG=/home/dropdir/log



#Locate all files that have been placed in directory within last 10 hours.
cd $INBOUND_DIR
for FILE in `find $INBOUND_DIR/file* -mmin -600`
        do
        #Compare the day the file was placed in directory with date on system.
        FILE_DATE=`ls -l $FILE | awk '{ print $6,$7}'`
        SYSTEM_DATE=`date | awk '{ print $2,$3 }'`

        if [ "$FILE_DATE" = "$SYSTEM_DATE" ]; then
                echo "Moving received $FILE to archive on $DATE" >> $LOG
                mv $FILE $ARCHIVE_DIR
                /bin/mail -s "$FILE was recieved on $DATE and archived" me@example.com < /dev/null
        else
                echo "No file received on $DATE" >> $LOG
                /bin/mail -s "$FILE was not received on $DATE and not archived" me@example.com < /dev/null
        fi
done
exit

Thanks

grail · 04-26-2010, 09:59 PM

I am not quite sure how effective the test is for date?
Assuming I use the test on my machine for file dummy.txt:

Code:

 
FILE_DATE="Apr 27" #this is after awk command
SYSTEM_DATE="Apr 27" #as i have run this right now

My issue is that should I run this script at say 1am and have had a file created within the last 10 hours (as per your find)
then I will be moved to the "else" as the dates are not equal??

Wouldn't it be better to check if the files that are less than 10 hours old exist in the ARCHIVE_DIR?
So maybe it could look like:

Code:

#!/bin/bash
#Description: Script to parse incoming file and archive if recent.


DATE=`date +%m-%d-%y`
INBOUND_DIR=/home/dropdir
ARCHIVE_DIR=/home/dropdir/archive
LOG=/home/dropdir/log

#Locate all files that have been placed in directory within last 10 hours.
#cd $INBOUND_DIR #this is not required if your find is already looking there

for FILE in `find $INBOUND_DIR/file* -mmin -600`
do
        if [[ ! -e "$ARCHIVE_DIR/${FILE##*/}" ]]; then
                echo "Moving received $FILE to archive on $DATE" >> $LOG
                mv $FILE $ARCHIVE_DIR
                /bin/mail -s "$FILE was recieved on $DATE and archived" me@example.com < /dev/null
        else
                echo "No file received on $DATE" >> $LOG
                /bin/mail -s "$FILE was not received on $DATE and not archived" me@example.com < /dev/null
        fi
done
exit

mrshanim · 04-26-2010, 10:06 PM

Hi,
I have some thaught.
Find the files with mtime or -mmin/-cmin 600 (here 600 consider it is variable as temp)
IF you are running your script after 10am or later on the day then use temp as 600. if you are running your script before 10 am say 8am, in this case you need to find the files which created in last 8hrs because first 2 hrs falls in previous day.
Then out temp variable will be 480.

Algo

//Find temp variable
if current time is >>10am temp=600
else find temp value by getting current days minutes or last two variables of the time command

For find all files with -mmin temp (or -cmin temp)
move each file to target dir
end of for

Regards
Manish

tuxtutorials · 04-27-2010, 03:32 PM

@grail regarding the tests for the date, the script would run at 10PM on the same day as the file being dropped. We usually get the file around 5PM on the same day. The logic behind this was that it compares the file with the current system date and if they match archive. Otherwise ignore. We had issues though when the cron job does not run on time and we have stray files dropped into the directory and files would not get archived. So my solution was to add the find command to only grab files 10 hours old. Not really sure if that test is necessary now since the find syntax only grabs files within the last 10 hours. I will take your logic for checking files in the archive directory and play with it.

Thanks for the input.

catkin · 04-28-2010, 03:25 AM

A potential issue with this asynchronous file drop and copy procedure is copying files before they have been fully dropped. The most robust solution is for the dropper to drop the fie with a special name (example *.part) and to rename it on completion of the drop. If that is not possible, a less robust solution is for the copy program to note the length of the file, pause a while and check the length again, only copying when the length has not changed.