LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-13-2011, 09:26 PM   #1
captainentropy
Member
 
Registered: Mar 2010
Location: Berkeley
Distribution: Ubuntu, Mint, CentOS
Posts: 74

Rep: Reputation: 0
overwriting files that are being operated on


Hi,

I'm not sure if this is a dumb question or not. I can't find anything that answers it (unless I'm not phrasing the question correctly).

Let's say I have a file that I need to modify somehow (sed, awk, sort, whatever) but I don't want to redirect the output to another file. How can I just overwrite the file with the new data, if that makes sense.

e.g. $sed '/X/d' file.txt > file.txt -- this just creates an empty file with the name file.txt
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-13-2011, 09:37 PM   #2
sag47
Senior Member
 
Registered: Sep 2009
Location: Philly, PA
Distribution: Kubuntu x64, RHEL, Fedora Core, FreeBSD, Windows x64
Posts: 1,431
Blog Entries: 33

Rep: Reputation: 358Reputation: 358Reputation: 358Reputation: 358
Code:
sed -i file.txt
-i for inline editing.

You could also have sed automatically make a backup copy like file.txt.bak.

Code:
sed -i.bak file.txt
See the man page for more information.

SAM
 
Old 09-13-2011, 09:47 PM   #3
captainentropy
Member
 
Registered: Mar 2010
Location: Berkeley
Distribution: Ubuntu, Mint, CentOS
Posts: 74

Original Poster
Rep: Reputation: 0
that did it thanks!
 
Old 09-14-2011, 02:58 AM   #4
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,755

Rep: Reputation: 466Reputation: 466Reputation: 466Reputation: 466Reputation: 466
It's worth noting that 'sed -i' does exactly what the OP thinks can be avoided -it simply creates a temporary which gets edited and then renamed/moved to the original file name. The only other way would be to read the files full contents into memory, eidt it there and then write it back out to the original file.
This:
file.txt > file.txt
doesn't work because as soon as the '>' redirect starts it truncates the original file, so there is nothing left to read there by sed(or others)
 
Old 09-14-2011, 08:06 AM   #5
ta0kira
Senior Member
 
Registered: Sep 2004
Distribution: FreeBSD 9.1, Kubuntu 12.10
Posts: 3,078

Rep: Reputation: Disabled
Quote:
Originally Posted by captainentropy View Post
e.g. $sed '/X/d' file.txt > file.txt -- this just creates an empty file with the name file.txt
The shell processes > file.txt before sed runs, meaning the file is truncated before sed even reads it.
Kevin Barry

Last edited by ta0kira; 09-14-2011 at 08:08 AM.
 
Old 09-14-2011, 08:34 AM   #6
cristalp
Member
 
Registered: Aug 2011
Distribution: Linux Mint
Posts: 103

Rep: Reputation: Disabled
Quote:
Originally Posted by captainentropy View Post
Hi,

I'm not sure if this is a dumb question or not. I can't find anything that answers it (unless I'm not phrasing the question correctly).

Let's say I have a file that I need to modify somehow (sed, awk, sort, whatever) but I don't want to redirect the output to another file. How can I just overwrite the file with the new data, if that makes sense.

e.g. $sed '/X/d' file.txt > file.txt -- this just creates an empty file with the name file.txt
Actually, I have just the same question for AWK. Anyone has an answer for awk??? Many Thanks!!
 
Old 09-14-2011, 10:52 AM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
You aren't going to find many shell programs that can be used to directly edit a file. Almost everything requires going through some form of temporary storage. So as a general rule of thumb, never use the same file for both input ad output.

An exception is an actual text editor. But only a few have the ability to be scripted externally. One of these is ed. See here: http://wiki.bash-hackers.org/howto/edit-ed I haven't done much with it myself yet.

I seem to recall that vi/vim can also be scripted, but I'm even less familiar with that.

Just out of curiosity, why is this so important to you? What's so bad or difficult about using temp files?


Edit: As long as the amount of text isn't too large, another option could be to store the command's output in a shell variable, instead of a file, then use that to overwrite the original afterwards.
Code:
outputvar=$( sed 's/foo/bar/' inputfile )
echo "$outputvar" >inputfile
unset outputvar

Last edited by David the H.; 09-14-2011 at 11:00 AM. Reason: addendum
 
2 members found this post helpful.
Old 09-14-2011, 01:07 PM   #8
ta0kira
Senior Member
 
Registered: Sep 2004
Distribution: FreeBSD 9.1, Kubuntu 12.10
Posts: 3,078

Rep: Reputation: Disabled
Quote:
Originally Posted by David the H. View Post
You aren't going to find many shell programs that can be used to directly edit a file. Almost everything requires going through some form of temporary storage. So as a general rule of thumb, never use the same file for both input ad output.

An exception is an actual text editor. But only a few have the ability to be scripted externally. One of these is ed. See here: http://wiki.bash-hackers.org/howto/edit-ed I haven't done much with it myself yet.
To elaborate, an important factor is the source of input. Most Unix command-line utilities read input data serially so that pipes can be used. If the input is to be overwritten by the output, the entire file must be read up front in case there are insertions or deletions. This would require the internal data representation to be different, and therefore also the process, for in-place modification vs. stream modification.
Kevin Barry
 
1 members found this post helpful.
Old 09-14-2011, 10:09 PM   #9
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,516

Rep: Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896Reputation: 1896
As you asked for awk:
Code:
awk '!/X/{print > FILENAME}' file.txt
 
Old 09-15-2011, 02:18 AM   #10
Ramurd
Member
 
Registered: Mar 2009
Location: Rotterdam, the Netherlands
Distribution: Slackwarelinux
Posts: 552

Rep: Reputation: 75
There is a handy utility for these kind of issues; it's called sponge; it's part of "moreutils"
Manpage: http://linux.die.net/man/1/sponge
I think this is the site: http://kitenet.net/~joey/code/moreutils/
 
1 members found this post helpful.
Old 09-15-2011, 03:29 AM   #11
cristalp
Member
 
Registered: Aug 2011
Distribution: Linux Mint
Posts: 103

Rep: Reputation: Disabled
Thanks!

Quote:
Originally Posted by David the H. View Post
You aren't going to find many shell programs that can be used to directly edit a file. Almost everything requires going through some form of temporary storage. So as a general rule of thumb, never use the same file for both input ad output.

An exception is an actual text editor. But only a few have the ability to be scripted externally. One of these is ed. See here: http://wiki.bash-hackers.org/howto/edit-ed I haven't done much with it myself yet.

I seem to recall that vi/vim can also be scripted, but I'm even less familiar with that.

Just out of curiosity, why is this so important to you? What's so bad or difficult about using temp files?


Edit: As long as the amount of text isn't too large, another option could be to store the command's output in a shell variable, instead of a file, then use that to overwrite the original afterwards.
Code:
outputvar=$( sed 's/foo/bar/' inputfile )
echo "$outputvar" >inputfile
unset outputvar

Hi David, Thanks for your detailed and excellent explanations! Only interesting for me to do this is that I have another software to call which needs to read several input files with common prefix. I need to edit one of them and keep it in the same way. Yes, I can save it in a temp and then move back it.

Yes you are right. Indeed, it is not so important at all and also not difficult at all to use a temp. I just guess, may be, to edit it in-place may make scripting more compact. I think the running speed would not increase as in-place editing always need the external editor. As I am really fresh in scripting, my ideas might be not really reasonable and looks naive.

As you advised, never use the same file for both input and output as a rule of thumb. I think it is a good suggestion. I would do it in the more standard way in my future scripting. Thanks a lot for your kind suggestion!
 
Old 09-19-2011, 10:37 AM   #12
cristalp
Member
 
Registered: Aug 2011
Distribution: Linux Mint
Posts: 103

Rep: Reputation: Disabled
Here is another answer for awk solution:

http://www.unix.com/shell-programmin...line-edit.html
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsync keeps overwriting some unmodified files figure002 Linux - Software 4 01-08-2010 01:16 PM
How to copy files without overwriting? Stephan_Craft Linux - Newbie 7 02-17-2009 05:28 AM
move files without overwriting files nathan2225 Linux - General 1 04-11-2008 04:01 PM
Please Help! Ubuntu keeps overwriting system files! ckr Ubuntu 2 07-13-2006 05:51 PM
php/ftp overwriting files.. prob. collen Programming 1 03-14-2005 04:56 AM


All times are GMT -5. The time now is 05:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration