LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-20-2011, 05:03 AM   #1
gary_in_springhill
Member
 
Registered: Mar 2008
Posts: 136

Rep: Reputation: 21
bash checking for file write completion before acting?


I have a bash script that checks for contents in a folder every 15 seconds and then acts on it's contents. This works great for the average size file however on very large files it starts acting on the file before it's completely written. Is there a facility in bash shell to get a file complete signal or such?
here is trigger to launch a larger script.
Code:
#!/bin/sh
while true
do
    $HOME/bin/hpgl.sh >/dev/null 2>&1 &
    sleep 15
done
##here is the section that when triggered gets run I only included the part that checks for a .ps file if no .ps it exits. Also what is to prevent on 
very large files as it's being written maybe 45 seconds the trigger launching the script multiple times?

for f in *.ps
do
if [ -f "$f" ]; then
JOBNAME="$(ls *ps)"
export JOBNAME
Thanks
 
Old 01-20-2011, 05:32 AM   #2
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi,

I don't see how the while loop is ever going to be exited. So the 'acting' part is never going to be executed. I assume that the 'acting' on the file actually happens inside the while loop. I further assume that the file that is going to be acted upon is generated by $HOME/bin/hpgl.sh. If you want to be sure that the file has definitely been written out then one way would be to not have $HOME/bin/hpgl.sh run in the background. Is there any reason why you can't do it like
$HOME/bin/hpgl.sh >/dev/null 2>&1

Also:
JOBNAME="$(ls *ps)"

That's probably not what you want. You already have the filename stored in $f.
 
1 members found this post helpful.
Old 01-20-2011, 05:54 AM   #3
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
I would use 'stat -c %s' to check that the file's size is constant. Check it once, wait, check it again, if it's the same, it's done.

Also, more info would help, because I don't know what is going on in the script.
 
1 members found this post helpful.
Old 01-20-2011, 07:17 AM   #4
gary_in_springhill
Member
 
Registered: Mar 2008
Posts: 136

Original Poster
Rep: Reputation: 21
script

The script is over 500 lines mostly dealing in totally unrelated calc's and such.
I didn't want to post unneeded data...
It triggers and works great now and is constantly running every few seconds as stated , the .ps file gets generated by an outside app that saves it in the monitored folder.

The stat command is exactly what I'm looking for

Thanks again
 
Old 01-20-2011, 07:46 AM   #5
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Stat will usually work, but is not bulletproof. gs often writes in bursts, and it's difficult to say if it's ready or just constructing a page heavy with graphics.

If you have the lsof package installed, lsof -t file-or-directory will output the PIDs of any processes that have the thing still opened. It does a lot of unnecessary work, so I don't recommend this either for this purpose. (It's excellent for checking if somebody has a configuration file open and so on, though.)

The best approach is to use the inotifywait command from the inotify-tools package. You simply run the command in monitoring mode, and it tells you when something interesting happens to the files or directories specified. For example:
Code:
#!/bin/bash
directories=( "one/" "another/" "third/" )
inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do
    if [ -f "$thing" ]; then

        echo "A new or modified file: $thing"

    fi
done
This will detect whenever a file is moved into one of the directories, and when a program that had a file open for writing, closes it.

You can also use inotifywait in a one-shot mode, with or without a timeout. It's all quite well described in the inotifywait manpage. Just remember that these are notifications of events already happened. Something else may already have happened to the target since.

Hope you find this useful,
Nominal Animal

Last edited by Nominal Animal; 03-21-2011 at 06:12 AM.
 
1 members found this post helpful.
Old 01-20-2011, 11:20 AM   #6
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,757

Rep: Reputation: 468Reputation: 468Reputation: 468Reputation: 468Reputation: 468
You might use 'lsof' or 'fuser' to check if the file is being used before attempting to process it.
 
1 members found this post helpful.
Old 01-20-2011, 01:12 PM   #7
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
The only robust solution is for whatever is writing the file to write it with a special name, say foo.part, and rename it, say to foo, when writing is completed.

Last edited by catkin; 01-20-2011 at 01:13 PM. Reason: added missing ,
 
1 members found this post helpful.
Old 01-20-2011, 05:56 PM   #8
gary_in_springhill
Member
 
Registered: Mar 2008
Posts: 136

Original Poster
Rep: Reputation: 21
tried a few solutions

I tried a few solutions:

As suspected stat only reports the changed file size at write complete.
Inotify wait won't monitor for one non-existent file.(can't monitor the whole folder due to other file activity and need *.ps *.pdf *.eps *.plt for files monitored, trouble)

solved it by adding a few seconds to the wait!
and recommending saving as pdf on very large files (these will be approx .1% of jobs anyway)
pdf's save 20 times faster from inkscape than ps files if fills are involved.

Thanks for all your help.
 
Old 01-21-2011, 12:57 AM   #9
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Quote:
Originally Posted by gary_in_springhill View Post
As suspected stat only reports the changed file size at write complete.
Inotify wait won't monitor for one non-existent file.(can't monitor the whole folder due to other file activity and need *.ps *.pdf *.eps *.plt for files monitored, trouble)
Sure you can. Just remember it will not monitor any subdirectories. Even if there are several tens of files moved or created per second, this will both reliable and lightweight:
Code:
#!/bin/bash
directories=( "one/" "another/" "third/" )
inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do
    [ -f "$thing" ] || continue
    case "$thing" in

        *.[Pp][Ss]|*.[Pp][Dd][Ff]|*.[Ee][Pp][Ss]|*.[Pp][Ll][Tt])

            echo "Do something to $thing"

            ;;
    esac
done
If you wish to filter by file type, you can use
Code:
#!/bin/bash
directories=( "one/" "another/" "third/" )
inotifywait -q --monitor --event close_write,moved_to --format '%w%f' "${directories[@]}" | while read thing ; do
    [ -f "$thing" ] || continue

    case "`file "$thing" 2>/dev/null | sed -ne 's|^.*:[\t ][\t ]*||p'`" in

        PDF*|Postscript*|Claris*)

            echo "Do something to $thing"

            ;;
    esac
done
instead. (I assumed your *.plt files were Claris Works files.) file only peeks at the beginning of the files, so it's not a heavy operation.

@catkin: Not true. Inotify is just as robust, and simpler to implement. It will notify reliably after the file has been closed for writing.
Nominal Animal

Last edited by Nominal Animal; 03-21-2011 at 06:09 AM.
 
1 members found this post helpful.
Old 01-21-2011, 09:46 AM   #10
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by Nominal Animal View Post
@catkin: Not true. Inotify is just as robust, and simpler to implement. It will notify reliably after the file has been closed for writing.
Thanks for the correction Nominal Animal
 
Old 01-22-2011, 06:43 AM   #11
gary_in_springhill
Member
 
Registered: Mar 2008
Posts: 136

Original Poster
Rep: Reputation: 21
works!

Works like dream even on inkscape ps files that take a minute to write!

Thanks VERY much

Can someone help write a small python method for me maybe 30-50 lines?
It involves writing and reading specific lines to and from a file which is beyond my meager experience
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How write a literal bash command in a bash file? xeon123 Linux - Newbie 6 11-29-2010 12:05 PM
[SOLVED] Bash, when I read sort write md5 file, the writen file have a space missing peter1234 Linux - General 3 09-21-2010 09:04 AM
Bash - is it possible to write to memory rather than a file basildon Linux - Newbie 7 09-24-2008 12:07 PM
best way to write a dynamic conf file in bash dkrysak Linux - General 6 02-13-2007 12:49 PM
bash - Checking for file existance? rignes Programming 5 02-18-2006 03:02 PM


All times are GMT -5. The time now is 07:16 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration