LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-26-2009, 12:48 PM   #1
yara
LQ Newbie
 
Registered: Oct 2009
Location: Nottingham, UK
Distribution: Suse, Fedora and Ubuntu
Posts: 5

Rep: Reputation: 0
replace several lines in a file with other lines in another file if condition


Hi,
I have this massive table file with some data in it and I want to replace some lines that are wrong with the correct ones that are in another table file of the same format. The wrong lines are not all together in a block but randomly distributed so I need to make a loop checking if the line is in the other file and if it is, replace it. I want to try and do it with sed or awk but I don't really know how to....
Can anyone help me out please?

Cheers
Yara
 
Old 10-26-2009, 01:47 PM   #2
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
http://www.linuxquestions.org/questi...script-212983/
 
Old 10-26-2009, 02:05 PM   #3
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301
Well, we can probably help, but maybe posting an example would help us do that.

Is it a whole line or just part of a line ? Note that if this table is truly "massive" then we should probably think about efficiency.
 
Old 10-26-2009, 02:18 PM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,520

Rep: Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944
Quote:
Originally Posted by yara View Post
Hi,
I have this massive table file with some data in it and I want to replace some lines that are wrong with the correct ones that are in another table file of the same format. The wrong lines are not all together in a block but randomly distributed so I need to make a loop checking if the line is in the other file and if it is, replace it. I want to try and do it with sed or awk but I don't really know how to....
Can anyone help me out please?

Cheers
Yara
Sure, we can help you. You'll have to do two things:

- Post what you've written so far, and tell us where you're having a problem.
- Give us a sample of the data you HAVE and tell us what you WANT

I doubt anyone is going to write a script for you, though, but we'll all be glad to help you debug what you've got.
 
Old 10-27-2009, 06:03 AM   #5
yara
LQ Newbie
 
Registered: Oct 2009
Location: Nottingham, UK
Distribution: Suse, Fedora and Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
Thank you for your answers!

My table is not THAT massive but it has many columns (53) and lines (1014) and I want to take anther table of the same number and format of columns but with fewer lines (just 137) and put those lines in my first table (replacing some lines), how will I know which lines I want to replace? well, they share the same name in column 1 (e.g. cl1018___m05_fit_s08_1_stamp_4861) but the other numbers (i.e. the other columns) are different so I want to just remove all the old line and put the new one.
I don't have a bash script written for this, I was trying in IDL but it's not working. I hope it's doable in a few lines, if it is not trivial that's, ok, I'll find another way

Thanks a lot,

Yara



Example of the 1st 3 lines of one of the tables (they are both looking the same but one has more lines than the other). I've highlited the beggining of the lines (1st column):

cl1018___m05_fit_s08_1_stamp_4861 1089.4197998 1089.4197998 1089.4197998 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 -125.91039313 -137.053852431 -116.2465474 5.0 5.0 5.0 1.5 1.5 1.5 0.453999996185 0.45
3999996185 0.453999996185 5.34110601835 2.16520130538 10.3841541184 -1.0 -1.0 -1.0 0.0 0.0 0.0 -3.97899657223
-4.02717979446 -3.92888928306 0.810829388979 0.752449589803 0.897580092026 839.755 779.753 60.001 3.184 0.684 2.500 1.376
6.615 1.988 0.445399999619 50.2599983215 14.1190003815
cl1018___m05_fit_s08_1_stamp_4959 167.127182007 167.127182007 167.127182007 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 -23.8006883376 -194.357654409 174.338860933 5.0 5.0 5.0 1.5 1.5 1.5 0.453999996185 0.45
3999996185 0.453999996185 130.247703812 40.8254399841 279.340202738 -1.0 -1.0 -1.0 0.0 0.0 0.0 -2.4028199259
-4.4395121732 -0.508937049938 1.69154484926 -0.190660485254 3.46142488923 748.328 724.328 24.000 2.615 0.615 2.000 1.286
4.981 0.166 0.445399999619 50.2599983215 14.1190003815
cl1018___m05_fit_s16_1_stamp_4861 1419.81530762 1419.81530762 1419.81530762 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 64.1375737567 47.0379275694 84.4564702671 5.0 5.0 5.0 1.5 1.5 1.5 0.517999982834 0.51
7999982834 0.517999982834 7.28441744566 2.61304158434 14.7767572381 -1.0 -1.0 -1.0 0.0 0.0 0.0 5.23872973961
5.03548626071 5.53032978522 3.36352940798 3.27005537028 3.44357155305 1453.227 1427.775 25.452 2.660 0.842
1.818 1.357 10.013 0.519 0.510500013828 73.4899978638 -1.53100610352
 
Old 10-27-2009, 10:38 AM   #6
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
Quote:
Originally Posted by yara View Post
I want to take anther table of the same number and format of columns but with fewer lines (just 137) and put those lines in my first table (replacing some lines)
So Yara, you are inserting *ALL* of the 137 lines into the larger table, if I understand you.

Quote:
... I want to just remove all the old line and put the new one.
Are you actually deleting the lines that begin with cl1018___m05_fit_s08_1_stamp_4861? I can't tell if you are replacing values in those lines, or wiping them out and inserting the other 137 lines as replacements.

The lines you posted tell us what you start with, but as TBOne and H_TeXMeX_H said, what do you *WANT* as a result?

Last edited by wfh; 10-27-2009 at 10:40 AM.
 
Old 10-27-2009, 10:51 AM   #7
yara
LQ Newbie
 
Registered: Oct 2009
Location: Nottingham, UK
Distribution: Suse, Fedora and Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
OK sorry for not explaing myself correctly, it's a line replacement what I'm looking for
I want to read the 137 lines on my file2 into file1, look for the equivalent ones in file 1 and replace those lines witht he ones in file2 but in a way that it replaces lines in file 2.

Maybe it's easier to explain with an example:

file1:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 3 3 5 7 9 9 9....
name4 4 4 5 6 1 2 3....
name5 5 5 1 2 2 2 3....

file2:
name3 0 0 1 1 1 4 4....
name5 0 3 4 4 2 2 1....

SO I want to take the lines in file 2 and replace the equivalent ones in file1 (the ones with the same name - i.e. value in Column1) to get this output:
output_file:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 0 0 1 1 1 4 4....
name4 4 4 5 6 1 2 3....
name5 0 3 4 4 2 2 1....

(I've highlighted the changes in file1)

Can you see my point?

Thank you VERY much for your patience!

Yara
 
Old 10-27-2009, 11:04 AM   #8
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,520

Rep: Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944Reputation: 7944
Quote:
Originally Posted by yara View Post
OK sorry for not explaing myself correctly, it's a line replacement what I'm looking for
I want to read the 137 lines on my file2 into file1, look for the equivalent ones in file 1 and replace those lines witht he ones in file2 but in a way that it replaces lines in file 2.

Maybe it's easier to explain with an example:

file1:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 3 3 5 7 9 9 9....
name4 4 4 5 6 1 2 3....
name5 5 5 1 2 2 2 3....

file2:
name3 0 0 1 1 1 4 4....
name5 0 3 4 4 2 2 1....

SO I want to take the lines in file 2 and replace the equivalent ones in file1 (the ones with the same name - i.e. value in Column1) to get this output:
output_file:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 0 0 1 1 1 4 4....
name4 4 4 5 6 1 2 3....
name5 0 3 4 4 2 2 1....

(I've highlighted the changes in file1)

Can you see my point?

Thank you VERY much for your patience!

Yara
No problem...we get it now. So, how about posting what you've written so far, and where you're getting stuck??

We'll be glad to HELP you, but I doubt that anyone is going to write your script FOR YOU. If you don't know about bash programming, there are lots of tutorials on the web (Google has lots of links), that can help you.

If I had to write this, I'd look at the sort and uniq utilities....read their man pages.
 
Old 10-27-2009, 12:31 PM   #9
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Quote:
Originally Posted by yara View Post
Can you see my point?

Thank you VERY much for your patience!

Yara
see here for an example
 
Old 10-27-2009, 02:29 PM   #10
yara
LQ Newbie
 
Registered: Oct 2009
Location: Nottingham, UK
Distribution: Suse, Fedora and Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
That is just excellent!
Thank you so much!
 
Old 10-27-2009, 03:08 PM   #11
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301
Quote:
Originally Posted by ghostdog74 View Post
see here for an example
truly elegant solution, man
 
Old 10-27-2009, 03:22 PM   #12
wfh
Member
 
Registered: Sep 2009
Location: Northern California
Distribution: Ubuntu Debian CentOS RHEL Suse
Posts: 164

Rep: Reputation: 44
I must say, ghostdog74 has the more elegant solution. All I came up with was this:

Code:
#!/bin/sh

foo=file1
bar=file2

while read line1 # Loop through 'file1'
do
   string1=$( echo $line1 |awk '{print $1}' );

   while read line2 # Loop through 'file2'
   do
      string2=$( echo $line2 |awk '{print $1}' ); 
      if [ "$string1" = "$string2" ]; then
          outline=$line2;
          break
      fi
      outline=$line1
   done < $bar
   echo $outline
done < $foo

Last edited by wfh; 10-27-2009 at 03:47 PM.
 
Old 10-27-2009, 03:46 PM   #13
yara
LQ Newbie
 
Registered: Oct 2009
Location: Nottingham, UK
Distribution: Suse, Fedora and Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
Yes, ghostdog74's solution is indeed very elegant AND effective....

Thank you so much again!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Substitute specific lines with lines from another file rahmathullakm Programming 4 01-10-2009 05:47 AM
shell scripe replace lines in file. nenad_m Programming 5 11-27-2008 10:59 AM
Replace line in file with two lines?! Possible? eur0dad Linux - General 2 09-07-2006 02:15 PM
How can I use a shell script to add and replace lines in a file? abefroman Programming 10 12-27-2005 05:05 PM
Replace blank/almost blank lines in file Wynd Linux - General 3 01-27-2004 04:49 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 02:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration