LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-05-2010, 09:32 AM   #1
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
How to make a command write to the file it's reading from?


For example I want a file to be processed by sed, and then overwrite the file with sed's output. I would try this:

Code:
sed '<regex goes here>' myfile > myfile
but it doesn't work as expected, instead it empties the file (I am thinking that as the first byte comes out of sed, it overwrites the whole file and sed has nothing more to do). How can I make this work?

Last edited by MTK358; 01-05-2010 at 09:33 AM.
 
Old 01-05-2010, 09:38 AM   #2
nuwen52
Member
 
Registered: Feb 2009
Distribution: Debian, CentOS 5, Gentoo, FreeBSD, Fedora, Mint, Slackware64
Posts: 208

Rep: Reputation: 46
See if your version of sed has the -i option. What that is supposed to do it edit the file "in place". So, your command would be: sed -i '<regex goes here>' myfile
 
Old 01-05-2010, 10:05 AM   #3
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Original Poster
Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
That's good, but what if I use something other than sed that doesn't have an "in-place" option?
 
Old 01-05-2010, 10:13 AM   #4
slacker_et
Member
 
Registered: Dec 2009
Distribution: Slackware
Posts: 113

Rep: Reputation: 23
Is there a specific reason why you have to edit the original file ?
Why not have the script make a copy of the file, edit the copy, and then rename both files when done.

--ET
 
Old 01-05-2010, 10:19 AM   #5
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Quote:
Originally Posted by MTK358 View Post
That's good, but what if I use something other than sed that doesn't have an "in-place" option?
Is there the real possibility that you'll be using something other than sed (which has the -i option)?
Maybe provide us with something closer to a real example of what you plan to do, and we could give better advice.

Sed's -i option can create a backup file as well, before doing the editing:

Code:
sed -i.BAK '/regex/' myfile
where the .BAK can be whatever extension you want to tack onto the backup filename.

You may also be able to use AWK to read the file, do some processing, and use > to direct the result to a new file..

Sasha

PS -- another idea is to cat the entire original file into a variable, then echo the variable through the processing tool (sed, awk) and then > back to the original file. A warning though: I'm not sure if there's a limit to how much stuff you can stick into a single variable; it may be 65535 bytes, or some other limit. However, in the case that the file is too large for a single variable, then cat it line by line into an array, then process the array one by one and > back to the original file.

Last edited by GrapefruiTgirl; 01-05-2010 at 10:23 AM.
 
Old 01-05-2010, 10:30 AM   #6
slacker_et
Member
 
Registered: Dec 2009
Distribution: Slackware
Posts: 113

Rep: Reputation: 23
Quote:
Originally Posted by MTK358 View Post
That's good, but what if I use something other than sed that doesn't have an "in-place" option?
Is this similar to what you are asking ?
Code:
#!/bin/ksh



ls -lid ${0}*

sed '3a#echo blah blah blah' $0 >$0.new
chmod +x $0.new

mv $0 $0.sav
mv $0.new $0
ls -lid ${0}*

exit 0
--ET
 
Old 01-05-2010, 11:28 AM   #7
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Original Poster
Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
Quote:
Originally Posted by slacker_et View Post
Is this similar to what you are asking ?
Code:
#!/bin/ksh



ls -lid ${0}*

sed '3a#echo blah blah blah' $0 >$0.new
chmod +x $0.new

mv $0 $0.sav
mv $0.new $0
ls -lid ${0}*

exit 0
--ET
I don't understand that, I'm not very good with shell scripts.

I know that you can make a copy of the file, but in my situation you have to run multiple instances of editing programs over the same file to achieve the final relult, and that way you would have to have two copies and swap them every time, and then if you insert a command in the middle of the sequence you would have to tediously modify the rest of the file.
 
Old 01-07-2010, 09:47 AM   #8
slacker_et
Member
 
Registered: Dec 2009
Distribution: Slackware
Posts: 113

Rep: Reputation: 23
Quote:
Originally Posted by MTK358 View Post
I don't understand that, I'm not very good with shell scripts.

I know that you can make a copy of the file, but in my situation you have to run multiple instances of editing programs over the same file to achieve the final relult, and that way you would have to have two copies and swap them every time, and then if you insert a command in the middle of the sequence you would have to tediously modify the rest of the file.
No you wouldn't have to "have two copies" or "tediously modify the rest of the file".

What you (or rather your script/program doing the editing) would do is copy the existing (ie current) file to {filename}.new, modify the {filename}.new, mv {filename} to {filename}.sav, and then mv the {filename}.new to {filename}.
And to avoid more than one process attempting to modify the file at the same time. You can incorporate "lockfile" into your scripts.

This is essentially what the command "visudo" does with /etc/sudoers.

--ET
 
Old 01-07-2010, 01:34 PM   #9
kofucii
Member
 
Registered: May 2007
Location: Bulgaria
Distribution: Slackware, SCO Unix
Posts: 62

Rep: Reputation: 20
Code:
cat myfile | sed 'rexeg' > myfile
 
Old 01-07-2010, 02:53 PM   #10
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Original Poster
Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
Quote:
Originally Posted by kofucii View Post
Code:
cat myfile | sed 'rexeg' > myfile
So that will work because sed's modified output won't affect cat?

It still would be nice to have a command that would buffer the output and then write it all at the very end.
 
Old 01-07-2010, 03:12 PM   #11
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by kofucii View Post
Code:
cat myfile | sed 'rexeg' > myfile
Which OS is this working on?

Code:
$ cat silly
This is just a silly
little test silly file to
check whether silly sed and cat
silly do work in the way described
by kofucii  ...

$ cat silly | sed 's/silly/clever/g' > silly
$ cat silly
$
For me it bears the same result as
Code:
sed 's/silly/clever/g' silly > silly
... an empty file.

I can reproduce that on x86_64, z and SPARC with
Slackware, RHEL and Solaris 10.


Cheers,
Tink
 
Old 01-07-2010, 03:23 PM   #12
kofucii
Member
 
Registered: May 2007
Location: Bulgaria
Distribution: Slackware, SCO Unix
Posts: 62

Rep: Reputation: 20
Quote:
Originally Posted by Tinkster View Post
Which OS is this working on?

Code:
$ cat silly
This is just a silly
little test silly file to
check whether silly sed and cat
silly do work in the way described
by kofucii  ...

$ cat silly | sed 's/silly/clever/g' > silly
$ cat silly
$
For me it bears the same result as
Code:
sed 's/silly/clever/g' silly > silly
... an empty file.

I can reproduce that on x86_64, z and SPARC with
Slackware, RHEL and Solaris 10.


Cheers,
Tink
Execute it from a script, file name must be specified as a variable and It will work:
Code:
#!/bin/bash
FILE=myfile
cat $FILE | sed "s/foo/bar/g" > $FILE

Last edited by kofucii; 01-07-2010 at 03:26 PM.
 
Old 01-07-2010, 03:41 PM   #13
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by kofucii View Post
Execute it from a script, file name must be specified as a variable and It will work:
Code:
#!/bin/bash
FILE=myfile
cat $FILE | sed "s/foo/bar/g" > $FILE
Hmmmm

Code:
$ cat silly test.sh
This is just a silly
little test silly file to
check whether silly sed and cat
silly do work in the way described
by kofucii silly ...

#!/bin/bash
FILE=silly
cat $FILE | sed "s/foo/bar/g" > $FILE
$
$
$ sh test.sh 
$ cat silly        
$
Hmmmm ... nope. Well ... on one of the three platforms
it does what you say - on two it results in an empty
file. Interesting - have to see WHY :}

[edit]
Corrrection: on the 2nd and subsequent runs it produces an
empty file on all three platforms. Very interesting.[/edit]


Cheers,
Tink

Last edited by Tinkster; 01-07-2010 at 03:43 PM.
 
Old 01-07-2010, 03:49 PM   #14
kofucii
Member
 
Registered: May 2007
Location: Bulgaria
Distribution: Slackware, SCO Unix
Posts: 62

Rep: Reputation: 20
It's not a reliable method. When executed repeatedly, It can result blank files. My guess, is that the problem goes from piping, and not closing properly the bite streams.

Last edited by kofucii; 01-07-2010 at 03:51 PM.
 
Old 01-07-2010, 03:57 PM   #15
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Original Poster
Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
Just for fun I will try to make a little C program that will store up all it's input from stdin in an array, and when it gets an EOF it will dump everything it stored up and exit.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Script passwd command by reading passwords from a file redleg7 Linux - General 1 01-12-2009 10:57 PM
reading a Windows Write file newbiesforever Linux - Software 15 02-12-2008 04:59 PM
Reading a file and running a command for each line. mijohnst Linux - General 11 08-22-2005 07:18 PM
Pop3 Command stream end of file while reading line sesimonsen Linux - Networking 0 05-13-2005 01:29 PM
I accidentally deleted make file in /usr/local/bin, now cannot use make command.... Niceman2005 Linux - Software 2 11-17-2004 08:55 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:57 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration