LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-19-2007, 04:26 AM   #1
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Rep: Reputation: 16
moving a file while is being written


Hi again.

Well, this is rather a theoretical question and it's not urgent - I don't want to abuse anyone's time.

Let's say a file ("myfile", in this case) is open for writing from a shell, by something like :
Code:
cat [SOME-HUGE-FILE] > myfile
 - or -
echo [SOME-HUGE-LINE] > myfile
and in the same time it gets moved, by something like:
Code:
mv myfile myfile2
The idea is that the move happens after the writing has started and before the writing has ended. I know that moving (with "mv") only changes two entries in the filesystem inodes (and thus is fast) but I wonder what exactly would happen. Of course I can use a lot of workarounds, locks, and so on, it's just this particular case I'm interested in. I'm sorry I have no experience with C++ to know how open() or write() work.

Thanks for reading!
 
Old 03-19-2007, 04:58 AM   #2
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,499

Rep: Reputation: 355Reputation: 355Reputation: 355Reputation: 355
You can easily check by yourself, the ongoing process will keep on writing to the newly renamed file, assuming is stays on the same filesystem.

By the way, C++ knowledge is unrelated. Open and write are written in C, and the file system implementation is what matters here.
 
Old 03-19-2007, 05:04 AM   #3
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Great, thanks. These are actually very good news for me. I was afraid that the redirected output would be completely lost.

Thank you.
 
Old 03-20-2007, 11:04 AM   #4
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,287

Rep: Reputation: 173Reputation: 173
easy to test yourself:

xterm 1
cat - > file1
start typing a bit

xterm2
mv file1 file2

xterm1
ctrl-D

(same directory of course
 
Old 03-21-2007, 01:15 PM   #5
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Yep; it works. Thanks
 
Old 03-21-2007, 06:34 PM   #6
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
I'll post this here; no need to start another thread for the next question - which is related to this discussion.

Let's say I continuously redirect the output of some command to a file; be it "tail -f".

Code:
tail -f mainfile > outfile
From time to time, I need to take the outfile out of the way, and replace it with a newly created file. And, by all means, the tail process (or any other) should not be interrupted.
I tried redirecting to a symbolic lynk, and then to force the re-creation of that link, like this:
Code:
touch outfile1
ln -s outfile1 outfile-link
tail -f mainfile > outfile-link

-- and after some time --

touch outfile2
ln -sf outfile2 outfile-link
But, (you guessed), tail keeps writing to outfile1.
Maybe if I'll instantly truncate outfile1 after copying it to outfile2 would do, though copying takes time and there won't be some way of telling if any new line was appended to outfile1 between those two actions.

If anyone has any idea I'll be most grateful - I hope someone has some more knowledge about what happens underneath the filesystem than me; cheers.

Last edited by omnio; 03-21-2007 at 06:41 PM.
 
Old 03-22-2007, 12:27 AM   #7
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,499

Rep: Reputation: 355Reputation: 355Reputation: 355Reputation: 355
You can't do that when your output goes directly to a file. The solution is to use an intermediary program like rotatelogs or cronolog: http://httpd.apache.org/docs/2.0/pro...otatelogs.html http://cronolog.org/
 
Old 03-22-2007, 01:19 AM   #8
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Thanks, I didn't know about them.

Quote:
Originally Posted by jlliagre
You can't do that when your output goes directly to a file. The solution is to use an intermediary program.
Excellent idea! Then I'll just pipe the output to a while loop which reads it line by line and writes it to a file. Let's say:

Code:
middle() {
    while true
    do
        read MYLINE
        -- test some condition --
        echo $MYLINE >> outfile1 (or outfile 2)
    done
}

tail -f mainfile | middle

(or: tail -f mainfile >( middle ) 
I'm not sure which one will work).
So far in my tests the read+echo loops are fast (bash uses its built-ins) so there shouldn't be a major performance penalty or delay (provided that the test inside the loop is a simple lock check and not more time-consuming, like du or grep).

Thanks!

Last edited by omnio; 03-22-2007 at 01:45 AM.
 
Old 03-22-2007, 03:35 AM   #9
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,287

Rep: Reputation: 173Reputation: 173
if you are writing to log files you can use the
logger (1) utility which adds times etc. for you.

we have to accept some limitations.

I admit though I never use it myself
 
Old 03-22-2007, 03:50 AM   #10
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,287

Rep: Reputation: 173Reputation: 173
or even something silly like this:

Code:
LOGFILE=blah.log
name=`basename $0`
LOGGER()
{

   exec >> $LOGFILE
   date=`date`

   /usr/bin/echo "[$name:$date] $@"
}

alias echo=LOGGER  # this works in ksh not bash (ksh 1 bash 0)
LOGGER hello
 
Old 03-23-2007, 09:25 AM   #11
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Well, I'm not altering the log files, but your advices are always welcome.

I'm rather continuously watching / backing them up.
What kinda worries me is that if I watch them with tail -f, they will be checked for line additions one per second (this is the default). If one or more lines are added within this ONE second and a log rotating mechanism interferes - and moves the file (and having tail following the filename), these lines will be lost. So I probably should be using something like:
Code:
tail -f --follow=name --sleep-interval=0
Ok, this should check the file continuously but I don't see any overburden so far.
Just a thought - maybe I sound paranoid. Thanks.
 
  


Reply

Tags
move, shell, write


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
cannot open file to written venki Linux - Newbie 5 12-18-2006 06:57 AM
help on compiling a file written in expect bluelite28 Linux - Software 2 06-27-2006 01:08 AM
reading file written by shell using C kapil_dev Programming 9 09-10-2005 12:11 AM
moving a file dominant Linux - Newbie 4 03-31-2004 06:39 AM
How to get last file written to directory kd5giv Linux - Networking 4 03-29-2004 02:26 PM


All times are GMT -5. The time now is 04:41 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration