Is there a generic solution to modifying files in place?
 

Is there are generic solution to modifying files in place?

For example, suppose I've got a file with data in it I need to process as follows:

which would search for lines of data with a specific prefix, chop off the prefix and write the results to standard out.

Now suppose I want to write the results back to the original file. What I can't do is:

The result is always an empty file because bash is unable to reading and writing a file at the same time.

The obvious solution is to write to a temporary file and to copy it back, which is bit ugly. A more elegant solution would be some sort of buffering command that will

* Read ALL of the original file
* Close the file
* Write the contents back to standard out.

The bash solution would be:


Does such a command exist?
Is there a better solution?

What you call a "buffering command" would actually write a temporary file...

For example, the following SED command replaces all occurences of "yes" to "no" and writes back to the same file. But what it REALLY does is first make a temporary file.

sed -i 's/yes/no/g' filename

some utilities also have options to write a backup file as part of the process.

OK that's really 3 commands on one line....

EDIT: I tried making a script to pipe in... but it appeared to have the same problem.

Thanks for the feed back.

I knew that a few commands had the ability to modify files (sed --in-place being a great example), but I can see this being a more general issue. This is why I used an example that was a bit more complex.

I don't mind commands that internally create temporary files (as sed does) as long as they clean up afterwards. These commands are certainly more elegant then managing the temp files myself.

The thing is, most commands don't have an in-place flag, so what to do in these cases? It looks like managing the file my self (with lumak's suggestion of the `tempfile` command) is the only real option, which is strange as I would imaging the to be a fairly common problem.

It seems strange until you realise that most of the text utils in unix were designed to talk to stdin/out so they could be used as inputs to each other, like links in a chain.

Bash like any other shell is perfectly able to read and write to a file at the same time. It does what you ask for and here you ask for truncating the file so it just does it.
Your suggestion won't work either, MyFile would be erased before any buffering take place anyway. The shell doesn't run the commands from left to right in a pipeline but the other way around. All but the rightmost program need the pipe listener party to be started before being launched.

Something like this should work though:
with Buffering command making sure it has finished reading its input before opening and writing to the file passed as parameter but not redirected to.

@ jlliagre
Thanks for that solution.
My question is then, why don't these commands already completely read and close before writing new files? would that cause issues with streaming to stdout? Would it only cause the pipe to respond slower? is there an existing command we can fake as a buffer? (e.g. somecommands MyFile | sed -i "s@@@" > MyFile)(that doesn't work but just illustrates my question)

Whether the commands do buffer or not doesn't matter. They haven't even a chance to run before it's too late. The shell itself handles the output redirection before launching any command. That's the reason why I suggest not to use ">" in the rightmost command.

Hi jlliagre,

Knowing that the the commands are started in reverse (right to left) order explains what I'm seeing. MyFile is emptied by the last command (the > operator) before the first command has had chance to read it.

I'm curious, why does the shell launch the commands in this order?

I'd like to give your alternative suggestion a go:
Do you have a suggestion about what I can use as BufferingCommand?

It doesn't necessarily, you can assume all commands are launched in parallel. However, the shell manages redirections before launching commands using them. There would be no point to launch the last command if the output file can't be created/written to. If it can, then the shell need to empty the file before running the command.
As you want to avoid temporary files, one simple solution might be:
Note that this script would only properly handle text files, not binary ones.

Sorry, but I'm not sure I understood

Could you please explain.

Incidentally I tried using tee as a buffering command. The following works:

However

fails!

The variable a contains the the standard input then this variable is printed to the file passed as a parameter.
It is much more risky that my suggestion. There is no full buffering with tee.
Not sure. Possibly a timing issue.

You can use that construction:

Works perfectly :-)

Thanks

It also helps if know that

will do the same as

There is always more to learn!

Thanks again