LinuxQuestions.org - [SOLVED] bash checking for file write completion before acting?

- Programming (http://www.linuxquestions.org/questions/programming-9/)

- - bash checking for file write completion before acting? (http://www.linuxquestions.org/questions/programming-9/bash-checking-for-file-write-completion-before-acting-857474/)

bash checking for file write completion before acting?

I have a bash script that checks for contents in a folder every 15 seconds and then acts on it's contents. This works great for the average size file however on very large files it starts acting on the file before it's completely written. Is there a facility in bash shell to get a file complete signal or such?
here is trigger to launch a larger script.

Code:

#!/bin/sh

while true

do

    $HOME/bin/hpgl.sh >/dev/null 2>&1 &

    sleep 15

done

##here is the section that when triggered gets run I only included the part that checks for a .ps file if no .ps it exits. Also what is to prevent on 

very large files as it's being written maybe 45 seconds the trigger launching the script multiple times?



for f in *.ps

do

if [ -f "$f" ]; then

JOBNAME="$(ls *ps)"

export JOBNAME

Thanks

Hi,

I don't see how the while loop is ever going to be exited. So the 'acting' part is never going to be executed. I assume that the 'acting' on the file actually happens inside the while loop. I further assume that the file that is going to be acted upon is generated by $HOME/bin/hpgl.sh. If you want to be sure that the file has definitely been written out then one way would be to not have $HOME/bin/hpgl.sh run in the background. Is there any reason why you can't do it like
$HOME/bin/hpgl.sh >/dev/null 2>&1

Also:
JOBNAME="$(ls *ps)"

That's probably not what you want. You already have the filename stored in $f.

I would use 'stat -c %s' to check that the file's size is constant. Check it once, wait, check it again, if it's the same, it's done.

Also, more info would help, because I don't know what is going on in the script.

The script is over 500 lines mostly dealing in totally unrelated calc's and such.
I didn't want to post unneeded data...
It triggers and works great now and is constantly running every few seconds as stated , the .ps file gets generated by an outside app that saves it in the monitored folder.

The stat command is exactly what I'm looking for

Thanks again

Stat will usually work, but is not bulletproof. gs often writes in bursts, and it's difficult to say if it's ready or just constructing a page heavy with graphics.

If you have the lsof package installed, lsof -t file-or-directory will output the PIDs of any processes that have the thing still opened. It does a lot of unnecessary work, so I don't recommend this either for this purpose. (It's excellent for checking if somebody has a configuration file open and so on, though.)

The best approach is to use the inotifywait command from the inotify-tools package. You simply run the command in monitoring mode, and it tells you when something interesting happens to the files or directories specified. For example:

Code:

#!/bin/bash

directories=( "one/" "another/" "third/" )

inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do

    if [ -f "$thing" ]; then



        echo "A new or modified file: $thing"



    fi

done

This will detect whenever a file is moved into one of the directories, and when a program that had a file open for writing, closes it.

You can also use inotifywait in a one-shot mode, with or without a timeout. It's all quite well described in the inotifywait manpage. Just remember that these are notifications of events already happened. Something else may already have happened to the target since.

Hope you find this useful,

Nominal Animal

You might use 'lsof' or 'fuser' to check if the file is being used before attempting to process it.

The only robust solution is for whatever is writing the file to write it with a special name, say foo.part, and rename it, say to foo, when writing is completed.

tried a few solutions

I tried a few solutions:

As suspected stat only reports the changed file size at write complete.
Inotify wait won't monitor for one non-existent file.(can't monitor the whole folder due to other file activity and need *.ps *.pdf *.eps *.plt for files monitored, trouble)

solved it by adding a few seconds to the wait! :)
and recommending saving as pdf on very large files (these will be approx .1% of jobs anyway)
pdf's save 20 times faster from inkscape than ps files if fills are involved.

Thanks for all your help.

Quote:

Originally Posted by gary_in_springhill (Post 4232270)

As suspected stat only reports the changed file size at write complete.
Inotify wait won't monitor for one non-existent file.(can't monitor the whole folder due to other file activity and need *.ps *.pdf *.eps *.plt for files monitored, trouble)

Sure you can. Just remember it will not monitor any subdirectories. Even if there are several tens of files moved or created per second, this will both reliable and lightweight:

Code:

#!/bin/bash

directories=( "one/" "another/" "third/" )

inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do

    [ -f "$thing" ] || continue

    case "$thing" in



        *.[Pp][Ss]|*.[Pp][Dd][Ff]|*.[Ee][Pp][Ss]|*.[Pp][Ll][Tt])



            echo "Do something to $thing"



            ;;

    esac

done

If you wish to filter by file type, you can use

Code:

#!/bin/bash

directories=( "one/" "another/" "third/" )

inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do

    [ -f "$thing" ] || continue



    case "`file "$thing" 2>/dev/null | sed -ne 's|^.*:[\t ][\t ]*||p'`" in



        PDF*|Postscript*|Claris*)



            echo "Do something to $thing"



            ;;

    esac

done

instead. (I assumed your *.plt files were Claris Works files.) file only peeks at the beginning of the files, so it's not a heavy operation.

@catkin: Not true. Inotify is just as robust, and simpler to implement. It will notify reliably after the file has been closed for writing.

Nominal Animal

Quote:

Originally Posted by Nominal Animal (Post 4232561)

@catkin: Not true. Inotify is just as robust, and simpler to implement. It will notify reliably after the file has been closed for writing.

Thanks for the correction Nominal Animal :)

Works like dream even on inkscape ps files that take a minute to write!

Thanks VERY much

Can someone help write a small python method for me maybe 30-50 lines?
It involves writing and reading specific lines to and from a file which is beyond my meager experience :)