overwriting files that are being operated on
 

Hi,

I'm not sure if this is a dumb question or not. I can't find anything that answers it (unless I'm not phrasing the question correctly).

Let's say I have a file that I need to modify somehow (sed, awk, sort, whatever) but I don't want to redirect the output to another file. How can I just overwrite the file with the new data, if that makes sense.

e.g. $sed '/X/d' file.txt > file.txt -- this just creates an empty file with the name file.txt

-i for inline editing.

You could also have sed automatically make a backup copy like file.txt.bak.

See the man page for more information.

SAM

that did it thanks!

It's worth noting that 'sed -i' does exactly what the OP thinks can be avoided -it simply creates a temporary which gets edited and then renamed/moved to the original file name. The only other way would be to read the files full contents into memory, eidt it there and then write it back out to the original file.
This:
file.txt > file.txt
doesn't work because as soon as the '>' redirect starts it truncates the original file, so there is nothing left to read there by sed(or others)

The shell processes > file.txt before sed runs, meaning the file is truncated before sed even reads it.
Kevin Barry

Actually, I have just the same question for AWK. Anyone has an answer for awk??? Many Thanks!!

You aren't going to find many shell programs that can be used to directly edit a file. Almost everything requires going through some form of temporary storage. So as a general rule of thumb, never use the same file for both input ad output.

An exception is an actual text editor. But only a few have the ability to be scripted externally. One of these is ed. See here: http://wiki.bash-hackers.org/howto/edit-ed I haven't done much with it myself yet.

I seem to recall that vi/vim can also be scripted, but I'm even less familiar with that.

Just out of curiosity, why is this so important to you? What's so bad or difficult about using temp files?


Edit: As long as the amount of text isn't too large, another option could be to store the command's output in a shell variable, instead of a file, then use that to overwrite the original afterwards.

To elaborate, an important factor is the source of input. Most Unix command-line utilities read input data serially so that pipes can be used. If the input is to be overwritten by the output, the entire file must be read up front in case there are insertions or deletions. This would require the internal data representation to be different, and therefore also the process, for in-place modification vs. stream modification.
Kevin Barry

As you asked for awk:

There is a handy utility for these kind of issues; it's called sponge; it's part of "moreutils"
Manpage: http://linux.die.net/man/1/sponge
I think this is the site: http://kitenet.net/~joey/code/moreutils/

Thanks!
 

Hi David, Thanks for your detailed and excellent explanations! Only interesting for me to do this is that I have another software to call which needs to read several input files with common prefix. I need to edit one of them and keep it in the same way. Yes, I can save it in a temp and then move back it.

Yes you are right. Indeed, it is not so important at all and also not difficult at all to use a temp. I just guess, may be, to edit it in-place may make scripting more compact. I think the running speed would not increase as in-place editing always need the external editor. As I am really fresh in scripting, my ideas might be not really reasonable and looks naive.

As you advised, never use the same file for both input and output as a rule of thumb. I think it is a good suggestion. I would do it in the more standard way in my future scripting. Thanks a lot for your kind suggestion!

Here is another answer for awk solution:

http://www.unix.com/shell-programmin...line-edit.html