LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-03-2013, 04:42 PM   #1
atjurhs
Member
 
Registered: Aug 2012
Posts: 182

Rep: Reputation: Disabled
why can't I operate on a file and replace it


hi guys,

i want to do something that's simple but i can't figure out the right syntax, so far everything i've tried (except creating a temp file that i have to later rename) destroys the original file leaving it empty, here's what I'm trying to do.....

awk -F " " '{print $1, $2, $3, $48*(-1), $5}' file.dat > file.dat

so that the output file is a replacement (with the same name) as the input file, like i said, it works fine if i give it a different output file name but i don't want to do that

is there something easy?

Tabby
 
Old 01-03-2013, 04:52 PM   #2
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,923

Rep: Reputation: 278Reputation: 278Reputation: 278
Post Temp File

Generally, you have to use a temp file.
 
Old 01-03-2013, 04:59 PM   #3
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,396
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
If your task can be accomplished with sed or Perl, you can use the '-i' switch to edit the file 'in-place'.
--- rod.

Last edited by theNbomr; 01-03-2013 at 05:03 PM.
 
Old 01-03-2013, 05:45 PM   #4
atjurhs
Member
 
Registered: Aug 2012
Posts: 182

Original Poster
Rep: Reputation: Disabled


i have sed, but i knew the math and print part of this was so easy to do in awk so i wrote the one-liner in awk. maybe there's an easy way to do math and print in sed???

i know it's not exactly right but could i do something like....

awk -F " " '{print $1, $2, $3, $48*(-1), $5}' inputfile.dat > outputfile.dat | mv outputfile.dat inputfile.dat

thanks guys for your help!

Tabby
 
Old 01-03-2013, 07:08 PM   #5
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,396
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
I'm pretty sure math is out of range for sed, but I can offer a Perl one-liner (test without the -i switch, first):

Code:
perl -i -e 'while(<>){ @z=split; print "$z[0], $z[1], $z[2], ",$z[47]*-1,"$z[4]\n";}' inputfile.dat
--- rod.
 
Old 01-03-2013, 07:49 PM   #6
atjurhs
Member
 
Registered: Aug 2012
Posts: 182

Original Poster
Rep: Reputation: Disabled
hi Rod,

i'm REALLY a newbie to script writing and barely waddle thru awk sed and bash script and usually with help.

i know perl is really powerful and really with math stuff, but it looks so cryptic, idk....

thanks soooooo much for the script! i'll give it a go tomorrow....

Tabby
 
Old 01-03-2013, 08:22 PM   #7
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,124

Rep: Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346
Quote:
Originally Posted by theNbomr View Post
I'm pretty sure math is out of range for sed,
The sed language has been shown to be Turing-complete, so math is at least theoretically within its range. Whether that use of sed is practical, ..., well it would almost certainly be easier than coding that in Ook!
 
Old 01-03-2013, 08:24 PM   #8
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,396
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
I think if you examine it somewhat, you'll see that it quite closely resembles the Awk script. The key differences are:
  • Perl needs explicit looping constructs ( while(<>){ .... } )
  • Perl needs to explicitly split into fields ( 'split', and the default separator is whitespace, just like Awk )
  • Perl uses zero-based array indexing in contrast to the built-in Awk variables named with non-zero positive integers

Yes, Perl is cryptic to those who have not drunk the magical elixir....

--- rod.

Last edited by theNbomr; 01-03-2013 at 08:26 PM.
 
Old 01-03-2013, 08:30 PM   #9
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,396
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Quote:
Originally Posted by rknichols View Post
The sed language has been shown to be Turing-complete, so math is at least theoretically within its range. Whether that use of sed is practical, ..., well it would almost certainly be easier than coding that in Ook!
I'd much rather see the sed version than the Ook version. Really, that would be impressive, either by virtue of the usefulness of learning something new, or by the length someone might go to to accomplish it.
--- rod.
 
Old 01-03-2013, 08:53 PM   #10
AnanthaP
Member
 
Registered: Jul 2004
Location: Chennai, India
Distribution: UBUNTU 5.10 since Jul-18,2006 on Intel 820 DC
Posts: 813

Rep: Reputation: 195Reputation: 195
Quote:
awk -F " " '{print $1, $2, $3, $48*(-1), $5}' file.dat > file_temp.dat
rm file.dat
mv file_temp.dat file.dat
I think this is what was meant in post #2.

By the way, perl is readily available with ALL distros.

OK
 
Old 01-03-2013, 09:12 PM   #11
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,059

Rep: Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266
Quote:
Originally Posted by AnanthaP View Post
Quote:
awk -F " " '{print $1, $2, $3, $48*(-1), $5}' file.dat > file_temp.dat
rm file.dat
mv file_temp.dat file.dat
I think this is what was meant in post #2.
The rm is not necessary.

Quote:
Originally Posted by theNbomr View Post
I think if you examine it somewhat, you'll see that it quite closely resembles the Awk script. The key differences are:[list][*]Perl needs explicit looping constructs ( while(<>){ .... } )[*]Perl needs to explicitly split into fields ( 'split', and the default separator is whitespace, just like Awk )
Quote:
perlrun:
...
-a

turns on autosplit mode when used with a -n or -p. An implicit split command to the @F array is done as the first thing inside the implicit while loop produced by the -n or -p.
...
-n

causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed -n or awk:
Quote:
Originally Posted by theNbomr View Post
I'd much rather see the sed version than the Ook version. Really, that would be impressive, either by virtue of the usefulness of learning something new, or by the length someone might go to to accomplish it.
--- rod.
Web search turns up dc.sed.
 
Old 01-03-2013, 10:11 PM   #12
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,923

Rep: Reputation: 278Reputation: 278Reputation: 278
You don't want to blow away the original file usually right away, at least not until you test the result of the changes. Coreutils programs have some bizarre behaviors under certain conditions. For instance: if you overwrite the first 20 bytes of a file, using the dd command, the outfile will be 20 bytes, unless you use notrunc. Sed has issues with certain charaters used in file names. So, you really want to keep the original file until the result file is tested, and then rm the original, or however you want to do it. I usually use cat.
 
Old 01-04-2013, 08:44 AM   #13
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,654

Rep: Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255
The problem you are doing is reading a data file and outputting an update to the file.

When the input and output are the same file, then the output will modify the input... causing problems.

Tools like sed use a tmp file internally, and then do the equivalent of "mv tmp originalfilename".

Think about how updates occur. If the original file had:

Code:
a
b
c
And you want to update it by replacing b with bb. If you use the same filename for both input and output, what happens is:
Code:
a
bb
because the second b in your update replaces the newline at the end with a b, and then puts a newline after that. Then there is the newline from the former "c" line, which has newline,newline...

The only saving grace (for very small files) is that the system buffers (or the runtime library buffers) could hold the entire file in memory... and give you the illusion of a tmp file. That doesn't always work either (updates to a file go to the same system buffer as used in input... though if the input has already been read it isn't a problem).

This is the same problem as having two people edit a file simultaneously... the output will be whoever closes the file last...

There is also the problem of making data shorter (replacing bb b for instance). You might get a duplicated data... or other funny looking stuff. This is closely related to issues with random access files (usually opened read/write). It works with fixed length records.. but if you extend/shorten a record your file gets corrupted unless you also do something to compensate (like using a temp file).
 
Old 01-04-2013, 01:26 PM   #14
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 3,124

Rep: Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346Reputation: 1346
Quote:
Originally Posted by jpollard View Post
The problem you are doing is reading a data file and outputting an update to the file.

When the input and output are the same file, then the output will modify the input... causing problems.
It's worse than that. When you try to run something like
Code:
awk '......' file.dat >file.dat
The shell will open file.dat for output, truncating it to zero length, before awk is even invoked, and all awk will see is the empty input file.
 
Old 01-04-2013, 11:59 PM   #15
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Quote:
... So that the output file is a replacement (with the same name) as the input file, like i said, it works fine if i give it a different output file name but i don't want to do that

is there something easy?
As you mentioned, to store the output of some operation in input file again, though it can be done by combining two commands like first command plus mv.

But in order to do it in a one-liner command, you can use process substitution, as:-
Code:
(cat outfile <(command...infile))> infile
For example, if infile.txt has:
Code:
A A
B
C C
D D
E
F F
Then invoke following:-
Code:
(cat outfile.txt <(awk 'NF>=2 {print $0}' infile.txt))> infile.txt
And now infile.txt will have:-
Code:
A A
C C
D D
F F
So make a try on this.

Last edited by shivaa; 01-05-2013 at 12:03 AM. Reason: Link added
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to do search & replace on a text file--need to extract URLs from a sitemap file Mountain Linux - General 4 08-07-2015 11:52 AM
[SOLVED] replace matched pattern from 1st file into 2nd file using awk sopier Programming 6 12-13-2011 10:58 AM
[SOLVED] Root unable to chmod, chown or operate on specific file chochem Linux - Software 2 09-06-2009 03:19 AM
replace line in CSV file and rename file connected to that name wademac Linux - Newbie 3 07-15-2009 02:09 PM
How to operate socket with file interface wangjinyi Programming 4 11-28-2005 12:31 AM


All times are GMT -5. The time now is 02:28 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration