LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-03-2012, 10:12 AM   #1
vikggg
LQ Newbie
 
Registered: May 2012
Posts: 2

Rep: Reputation: Disabled
i/o redirection destn file (moved, renamed, edited), will there be data corruption ?


i have a script that is running as a background process,
redirects the stdout and stderr to some file as below.

script.py >> log/script.log 2>&1 &

Now while the script is running, i moved the log file, renamed it (since it was growing too big).

What i did observe was that even after renaming the file it was growing in size.

--- Question 1: Can somebody explain this behavior ? (my guess is the FD still remains unchanged)

Then I cleaned up some garbage messages in it and saved it again.

I was expecting the system to detect closed FD and create a new file and continue the I/O re-direction. (Learnt it the hard way that i was wrong)

--- Question 2: Saving the file now most probably changed the file handle information. What happens to the background process that has I/O redirection.. ?? does it still keep on writing to the FD info that it had from before ? Should I expect data corruption on disk ? How do i recover/repair without having to stop my script ?

Thanks,
Vik
 
Old 05-03-2012, 10:31 AM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
On most UNIX/Linux filesystems files have "names" and "inodes". It is the inode in which the data is actually stored. When you rename (mv) a file (on the same filesystem) you are simply putting a different name on the same inode. Any process that had the file "open" will continue to have it open because it is the inode rather than the name.

This causes a common problem in that people often delete (rm) a log file but because it is "open" only the name is deleted - the inode is still in place so no space is recovered.

You can run "lsof <filename>" to see if any process has the file "open".

The proper way to do this is to stop the process that has the file open, rename or delete the old file, create a new empty file with the old name then restart the process. After that if you chose to save the old file rather than delete it you typically want to compress (using something like gzip) the old file that you renamed so that it frees up more space on the filesystem but is available for review later if necessary. (Note that compression requires space for the old file AND the new compressed file until the compression is complete - sometimes you need to mv the file to another filesystem, do the compression then mv the compressed file back to original filesystem.)

Note that when you use "mv" to move a file from one filesystem to another you are actually copying from the inode original filesystem to a different inode on the new filesystem then removing the old inode.
 
Old 05-03-2012, 11:01 AM   #3
vikggg
LQ Newbie
 
Registered: May 2012
Posts: 2

Original Poster
Rep: Reputation: Disabled
lsof -p 27267

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
...
script.py 27267 root 1w REG 8,3 645903454 110985451 /home/scripts/logs/script.log~ (deleted)
script.py 27267 root 2w REG 8,3 645903454 110985451 /home/scripts/logs/script.log~ (deleted)
..

The script still running it is still going to shell out some logging information, with the destination file now marked deleted, what happens to all that stdout/stderr ? Hopefully it doesnt write it to the old inode (the disk might just grow in size, which eventually might get overwritten - data corrouption), or does it just throw the stdout/stderr redirection out of the window and move on ?

The reason i ask, is i dont want to stop the script and re-run it again (will loose 2-3 days of execution time)
 
Old 05-03-2012, 11:22 AM   #4
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
It DOES continue to write to the same inode. You can't change the stderr/stdout after the start of the process. Do NOT delete a file that is open. It does NOT free up space - it only makes it harder to find out what is using the space later. Once you delete such a file the only way to clear the inode is to stop the process that has it open. (This of course can also be done by a reboot because that will stop all processes.)

If you are running out of space and do NOT want to stop the process then your only option is to increase the size of the filesystem.
 
1 members found this post helpful.
Old 05-03-2012, 02:59 PM   #5
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
This won't help you currently, but in the future you have a couple of options. Rather than having the script write each message to stdout and then you piping that into a logfile on execution, you could have the script write each message to the logfile directly. That would allow you to move/rename the logfile as needed without having to stop the script. Alternatively, you could write a logger script that reads from stdin and writes to a rotating logfile itself. Then rather than running "script.py >> logfile", you would run "script.py | logger"
 
Old 05-04-2012, 10:48 AM   #6
war49
LQ Newbie
 
Registered: Feb 2012
Location: Indonesia
Distribution: Slackware, Centos, Debian, RHEL
Posts: 13

Rep: Reputation: Disabled
Quote:
Originally Posted by vikggg View Post

script.py >> log/script.log 2>&1 &
If you affraid the script.log will grow to be huge. May be you can use database software to store script.py output. Or, if you consistently using log file (script.log), i think it would be better you rotated the log file and gziped (example: /var/log/messages, /var/log/messages1.gz, /var/log/messages2.gz ..etc).

Last edited by war49; 05-04-2012 at 10:50 AM.
 
Old 05-04-2012, 11:57 AM   #7
snowmobile74
LQ Newbie
 
Registered: Nov 2003
Location: Reston, VA
Distribution: Slackware for everything
Posts: 22

Rep: Reputation: 1
Interesting, but what if you did this.

cp log/script.log log/script.log.2; echo "" > log/script.log
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] wget sent edited xml as post data deadeyes Linux - General 1 07-09-2010 08:28 AM
GNOME SlackBuild - Broadband icon moved and renamed ? slacker_et Slackware 1 12-31-2009 06:53 PM
data moved to some where i don't know! Mr.mick-duck Mandriva 2 03-26-2008 10:16 AM
how to know how much data is moved on network nephish Linux - Server 3 01-13-2008 07:17 PM
Data Corruption on file move? mijohnst Linux - Networking 1 12-22-2004 12:44 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:17 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration