replace several lines in a file with other lines in another file if condition

yara · 10-26-2009, 12:48 PM

Hi,
I have this massive table file with some data in it and I want to replace some lines that are wrong with the correct ones that are in another table file of the same format. The wrong lines are not all together in a block but randomly distributed so I need to make a loop checking if the line is in the other file and if it is, replace it. I want to try and do it with sed or awk but I don't really know how to....
Can anyone help me out please?

Cheers
Yara

wfh · 10-26-2009, 01:47 PM

http://www.linuxquestions.org/questi...script-212983/

H_TeXMeX_H · 10-26-2009, 02:05 PM

Well, we can probably help, but maybe posting an example would help us do that.

Is it a whole line or just part of a line ? Note that if this table is truly "massive" then we should probably think about efficiency.

TB0ne · 10-26-2009, 02:18 PM

Quote:

Originally Posted by yara

Hi,
I have this massive table file with some data in it and I want to replace some lines that are wrong with the correct ones that are in another table file of the same format. The wrong lines are not all together in a block but randomly distributed so I need to make a loop checking if the line is in the other file and if it is, replace it. I want to try and do it with sed or awk but I don't really know how to....
Can anyone help me out please?

Cheers
Yara

Sure, we can help you. You'll have to do two things:

- Post what you've written so far, and tell us where you're having a problem.
- Give us a sample of the data you HAVE and tell us what you WANT

I doubt anyone is going to write a script for you, though, but we'll all be glad to help you debug what you've got.

yara · 10-27-2009, 06:03 AM

Thank you for your answers!

My table is not THAT massive but it has many columns (53) and lines (1014) and I want to take anther table of the same number and format of columns but with fewer lines (just 137) and put those lines in my first table (replacing some lines), how will I know which lines I want to replace? well, they share the same name in column 1 (e.g. cl1018___m05_fit_s08_1_stamp_4861) but the other numbers (i.e. the other columns) are different so I want to just remove all the old line and put the new one.
I don't have a bash script written for this, I was trying in IDL but it's not working. I hope it's doable in a few lines, if it is not trivial that's, ok, I'll find another way

Thanks a lot,

Yara

Example of the 1st 3 lines of one of the tables (they are both looking the same but one has more lines than the other). I've highlited the beggining of the lines (1st column):

cl1018___m05_fit_s08_1_stamp_4861 1089.4197998 1089.4197998 1089.4197998 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 -125.91039313 -137.053852431 -116.2465474 5.0 5.0 5.0 1.5 1.5 1.5 0.453999996185 0.45
3999996185 0.453999996185 5.34110601835 2.16520130538 10.3841541184 -1.0 -1.0 -1.0 0.0 0.0 0.0 -3.97899657223
-4.02717979446 -3.92888928306 0.810829388979 0.752449589803 0.897580092026 839.755 779.753 60.001 3.184 0.684 2.500 1.376
6.615 1.988 0.445399999619 50.2599983215 14.1190003815
cl1018___m05_fit_s08_1_stamp_4959 167.127182007 167.127182007 167.127182007 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 -23.8006883376 -194.357654409 174.338860933 5.0 5.0 5.0 1.5 1.5 1.5 0.453999996185 0.45
3999996185 0.453999996185 130.247703812 40.8254399841 279.340202738 -1.0 -1.0 -1.0 0.0 0.0 0.0 -2.4028199259
-4.4395121732 -0.508937049938 1.69154484926 -0.190660485254 3.46142488923 748.328 724.328 24.000 2.615 0.615 2.000 1.286
4.981 0.166 0.445399999619 50.2599983215 14.1190003815
cl1018___m05_fit_s16_1_stamp_4861 1419.81530762 1419.81530762 1419.81530762 0.0 0.0 0.0 0.0 0.0 0.0 0.
0 0.0 0.0 64.1375737567 47.0379275694 84.4564702671 5.0 5.0 5.0 1.5 1.5 1.5 0.517999982834 0.51
7999982834 0.517999982834 7.28441744566 2.61304158434 14.7767572381 -1.0 -1.0 -1.0 0.0 0.0 0.0 5.23872973961
5.03548626071 5.53032978522 3.36352940798 3.27005537028 3.44357155305 1453.227 1427.775 25.452 2.660 0.842
1.818 1.357 10.013 0.519 0.510500013828 73.4899978638 -1.53100610352

wfh · 10-27-2009, 10:38 AM

Quote:

Originally Posted by yara

I want to take anther table of the same number and format of columns but with fewer lines (just 137) and put those lines in my first table (replacing some lines)

So Yara, you are inserting *ALL* of the 137 lines into the larger table, if I understand you.

Quote:

... I want to just remove all the old line and put the new one.

Are you actually deleting the lines that begin with cl1018___m05_fit_s08_1_stamp_4861? I can't tell if you are replacing values in those lines, or wiping them out and inserting the other 137 lines as replacements.

The lines you posted tell us what you start with, but as TBOne and H_TeXMeX_H said, what do you *WANT* as a result?

yara · 10-27-2009, 10:51 AM

OK sorry for not explaing myself correctly, it's a line replacement what I'm looking for
I want to read the 137 lines on my file2 into file1, look for the equivalent ones in file 1 and replace those lines witht he ones in file2 but in a way that it replaces lines in file 2.

Maybe it's easier to explain with an example:

file1:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 3 3 5 7 9 9 9....
name4 4 4 5 6 1 2 3....
name5 5 5 1 2 2 2 3....

file2:
name3 0 0 1 1 1 4 4....
name5 0 3 4 4 2 2 1....

SO I want to take the lines in file 2 and replace the equivalent ones in file1 (the ones with the same name - i.e. value in Column1) to get this output:
output_file:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 0 0 1 1 1 4 4....
name4 4 4 5 6 1 2 3....
name5 0 3 4 4 2 2 1....

(I've highlighted the changes in file1)

Can you see my point?

Thank you VERY much for your patience!

Yara

TB0ne · 10-27-2009, 11:04 AM

Quote:

Originally Posted by yara

OK sorry for not explaing myself correctly, it's a line replacement what I'm looking for
I want to read the 137 lines on my file2 into file1, look for the equivalent ones in file 1 and replace those lines witht he ones in file2 but in a way that it replaces lines in file 2.

Maybe it's easier to explain with an example:

file1:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 3 3 5 7 9 9 9....
name4 4 4 5 6 1 2 3....
name5 5 5 1 2 2 2 3....

file2:
name3 0 0 1 1 1 4 4....
name5 0 3 4 4 2 2 1....

SO I want to take the lines in file 2 and replace the equivalent ones in file1 (the ones with the same name - i.e. value in Column1) to get this output:
output_file:
name1 1 1 4 5 6 7 8....
name2 2 2 6 7 8 9 0....
name3 0 0 1 1 1 4 4....
name4 4 4 5 6 1 2 3....
name5 0 3 4 4 2 2 1....

(I've highlighted the changes in file1)

Can you see my point?

Thank you VERY much for your patience!

Yara

No problem...we get it now. So, how about posting what you've written so far, and where you're getting stuck??

We'll be glad to HELP you, but I doubt that anyone is going to write your script FOR YOU. If you don't know about bash programming, there are lots of tutorials on the web (Google has lots of links), that can help you.

If I had to write this, I'd look at the sort and uniq utilities....read their man pages.

ghostdog74 · 10-27-2009, 12:31 PM

Quote:

Originally Posted by yara

Can you see my point?

Thank you VERY much for your patience!

Yara

see here for an example

yara · 10-27-2009, 02:29 PM

That is just excellent!
Thank you so much!

H_TeXMeX_H · 10-27-2009, 03:08 PM

Quote:

Originally Posted by ghostdog74

see here for an example

truly elegant solution, man

wfh · 10-27-2009, 03:22 PM

I must say, ghostdog74 has the more elegant solution. All I came up with was this:

Code:

#!/bin/sh

foo=file1
bar=file2

while read line1 # Loop through 'file1'
do
   string1=$( echo $line1 |awk '{print $1}' );

   while read line2 # Loop through 'file2'
   do
      string2=$( echo $line2 |awk '{print $1}' ); 
      if [ "$string1" = "$string2" ]; then
          outline=$line2;
          break
      fi
      outline=$line1
   done < $bar
   echo $outline
done < $foo

yara · 10-27-2009, 03:46 PM

Yes, ghostdog74's solution is indeed very elegant AND effective....

Thank you so much again!