moving a file while is being written

omnio · 03-19-2007, 04:26 AM

Hi again.

Well, this is rather a theoretical question and it's not urgent - I don't want to abuse anyone's time.

Let's say a file ("myfile", in this case) is open for writing from a shell, by something like :

Code:

cat [SOME-HUGE-FILE] > myfile
 - or -
echo [SOME-HUGE-LINE] > myfile

and in the same time it gets moved, by something like:

Code:

mv myfile myfile2

The idea is that the move happens after the writing has started and before the writing has ended. I know that moving (with "mv") only changes two entries in the filesystem inodes (and thus is fast) but I wonder what exactly would happen. Of course I can use a lot of workarounds, locks, and so on, it's just this particular case I'm interested in. I'm sorry I have no experience with C++ to know how open() or write() work.

Thanks for reading!

jlliagre · 03-19-2007, 04:58 AM

You can easily check by yourself, the ongoing process will keep on writing to the newly renamed file, assuming is stays on the same filesystem.

By the way, C++ knowledge is unrelated. Open and write are written in C, and the file system implementation is what matters here.

omnio · 03-19-2007, 05:04 AM

Great, thanks. These are actually very good news for me. I was afraid that the redirected output would be completely lost.

Thank you.

bigearsbilly · 03-20-2007, 11:04 AM

easy to test yourself:

xterm 1
cat - > file1
start typing a bit

xterm2
mv file1 file2

xterm1
ctrl-D

(same directory of course

omnio · 03-21-2007, 01:15 PM

Yep; it works. Thanks

omnio · 03-21-2007, 06:34 PM

I'll post this here; no need to start another thread for the next question - which is related to this discussion.

Let's say I continuously redirect the output of some command to a file; be it "tail -f".

Code:

tail -f mainfile > outfile

From time to time, I need to take the outfile out of the way, and replace it with a newly created file. And, by all means, the tail process (or any other) should not be interrupted.
I tried redirecting to a symbolic lynk, and then to force the re-creation of that link, like this:

Code:

touch outfile1
ln -s outfile1 outfile-link
tail -f mainfile > outfile-link

-- and after some time --

touch outfile2
ln -sf outfile2 outfile-link

But, (you guessed), tail keeps writing to outfile1.
Maybe if I'll instantly truncate outfile1 after copying it to outfile2 would do, though copying takes time and there won't be some way of telling if any new line was appended to outfile1 between those two actions.

If anyone has any idea I'll be most grateful - I hope someone has some more knowledge about what happens underneath the filesystem than me; cheers.

jlliagre · 03-22-2007, 12:27 AM

You can't do that when your output goes directly to a file. The solution is to use an intermediary program like rotatelogs or cronolog: http://httpd.apache.org/docs/2.0/pro...otatelogs.html http://cronolog.org/

omnio · 03-22-2007, 01:19 AM

Quote:

Originally Posted by jlliagre

http://httpd.apache.org/docs/2.0/pro...otatelogs.html http://cronolog.org/

Thanks, I didn't know about them.

Quote:

Originally Posted by jlliagre

You can't do that when your output goes directly to a file. The solution is to use an intermediary program.

Excellent idea! Then I'll just pipe the output to a while loop which reads it line by line and writes it to a file. Let's say:

Code:

middle() {
    while true
    do
        read MYLINE
        -- test some condition --
        echo $MYLINE >> outfile1 (or outfile 2)
    done
}

tail -f mainfile | middle

(or: tail -f mainfile >( middle ) 
I'm not sure which one will work).

So far in my tests the read+echo loops are fast (bash uses its built-ins) so there shouldn't be a major performance penalty or delay (provided that the test inside the loop is a simple lock check and not more time-consuming, like du or grep).

Thanks!

bigearsbilly · 03-22-2007, 03:35 AM

if you are writing to log files you can use the
logger (1) utility which adds times etc. for you.

we have to accept some limitations.

I admit though I never use it myself

bigearsbilly · 03-22-2007, 03:50 AM

or even something silly like this:

Code:

LOGFILE=blah.log
name=`basename $0`
LOGGER()
{

   exec >> $LOGFILE
   date=`date`

   /usr/bin/echo "[$name:$date] $@"
}

alias echo=LOGGER  # this works in ksh not bash (ksh 1 bash 0)
LOGGER hello

omnio · 03-23-2007, 09:25 AM

Well, I'm not altering the log files, but your advices are always welcome.

I'm rather continuously watching / backing them up.
What kinda worries me is that if I watch them with tail -f, they will be checked for line additions one per second (this is the default). If one or more lines are added within this ONE second and a log rotating mechanism interferes - and moves the file (and having tail following the filename), these lines will be lost. So I probably should be using something like:

Code:

tail -f --follow=name --sleep-interval=0

Ok, this should check the file continuously but I don't see any overburden so far.
Just a thought - maybe I sound paranoid. Thanks.