Is there a generic solution to modifying files in place?
Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Is there a generic solution to modifying files in place?
Is there are generic solution to modifying files in place?
For example, suppose I've got a file with data in it I need to process as follows:
Code:
grep "^tag-" MyFile | cut -d"-" -f2
which would search for lines of data with a specific prefix, chop off the prefix and write the results to standard out.
Now suppose I want to write the results back to the original file. What I can't do is:
Code:
grep "^tag-" MyFile | cut -d"-" -f2 > MyFile
The result is always an empty file because bash is unable to reading and writing a file at the same time.
The obvious solution is to write to a temporary file and to copy it back, which is bit ugly. A more elegant solution would be some sort of buffering command that will
* Read ALL of the original file
* Close the file
* Write the contents back to standard out.
What you call a "buffering command" would actually write a temporary file...
For example, the following SED command replaces all occurences of "yes" to "no" and writes back to the same file. But what it REALLY does is first make a temporary file.
sed -i 's/yes/no/g' filename
some utilities also have options to write a backup file as part of the process.
I knew that a few commands had the ability to modify files (sed --in-place being a great example), but I can see this being a more general issue. This is why I used an example that was a bit more complex.
I don't mind commands that internally create temporary files (as sed does) as long as they clean up afterwards. These commands are certainly more elegant then managing the temp files myself.
The thing is, most commands don't have an in-place flag, so what to do in these cases? It looks like managing the file my self (with lumak's suggestion of the `tempfile` command) is the only real option, which is strange as I would imaging the to be a fairly common problem.
It looks like managing the file my self (with lumak's suggestion of the `tempfile` command) is the only real option, which is strange as I would imaging the to be a fairly common problem.
It seems strange until you realise that most of the text utils in unix were designed to talk to stdin/out so they could be used as inputs to each other, like links in a chain.
Distribution: Solaris 11.4, Oracle Linux, Mint, Debian/WSL
Posts: 9,789
Rep:
Quote:
Originally Posted by stormcloud
The result is always an empty file because bash is unable to reading and writing a file at the same time.
Bash like any other shell is perfectly able to read and write to a file at the same time. It does what you ask for and here you ask for truncating the file so it just does it.
Quote:
The obvious solution is to write to a temporary file and to copy it back, which is bit ugly. * Read ALL of the original file
* Close the file
* Write the contents back to standard out.
Does such a command exist?
Is there a better solution?
Your suggestion won't work either, MyFile would be erased before any buffering take place anyway. The shell doesn't run the commands from left to right in a pipeline but the other way around. All but the rightmost program need the pipe listener party to be started before being launched.
@ jlliagre
Thanks for that solution.
My question is then, why don't these commands already completely read and close before writing new files? would that cause issues with streaming to stdout? Would it only cause the pipe to respond slower? is there an existing command we can fake as a buffer? (e.g. somecommands MyFile | sed -i "s@@@" > MyFile)(that doesn't work but just illustrates my question)
Distribution: Solaris 11.4, Oracle Linux, Mint, Debian/WSL
Posts: 9,789
Rep:
Whether the commands do buffer or not doesn't matter. They haven't even a chance to run before it's too late. The shell itself handles the output redirection before launching any command. That's the reason why I suggest not to use ">" in the rightmost command.
Knowing that the the commands are started in reverse (right to left) order explains what I'm seeing. MyFile is emptied by the last command (the > operator) before the first command has had chance to read it.
I'm curious, why does the shell launch the commands in this order?
I'd like to give your alternative suggestion a go:
Distribution: Solaris 11.4, Oracle Linux, Mint, Debian/WSL
Posts: 9,789
Rep:
Quote:
Originally Posted by stormcloud
I'm curious, why does the shell launch the commands in this order?
It doesn't necessarily, you can assume all commands are launched in parallel. However, the shell manages redirections before launching commands using them. There would be no point to launch the last command if the output file can't be created/written to. If it can, then the shell need to empty the file before running the command.
Quote:
Do you have a suggestion about what I can use as BufferingCommand?
As you want to avoid temporary files, one simple solution might be:
Code:
a=$(cat)
echo "$a" > ${1:?}
Note that this script would only properly handle text files, not binary ones.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.