LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-31-2013, 11:43 AM   #1
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Rep: Reputation: Disabled
Question Shell Script to replace specific columns on matched lines


Hi,
I have a data file that has many lines in it, that is space delimited. What I want to do is match lines in the file that have the word host in it and then i want to replace columns 3-6 with data that I have in another file. The other file only has the information that belongs on those specific lines. Any ideas?
 
Old 01-31-2013, 11:55 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,601

Rep: Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570
Quote:
Originally Posted by axl718 View Post
Hi,
I have a data file that has many lines in it, that is space delimited. What I want to do is match lines in the file that have the word host in it and then i want to replace columns 3-6 with data that I have in another file. The other file only has the information that belongs on those specific lines. Any ideas?
Ideas? Yes...write a script to do what you want. You can easily write such a script using bash, perl, or any number of other languages. The cut, sed, awk, and grep commands will probably all be used.

There is a link at the bottom of this post to a bash scripting tutorial...that's a good place to start. If you have specific problems/questions, please post them (along with samples of the input/output data), and we will be glad to help.
 
Old 01-31-2013, 12:00 PM   #3
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
Well yeah I know write a script. I'm having trouble matching the lines and I'm having trouble replacing the columns. I am fairly new at this. I'm thinking awk or sed but I just don't know the right syntax to get it all working.
 
Old 01-31-2013, 12:33 PM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,601

Rep: Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570Reputation: 2570
Quote:
Originally Posted by axl718 View Post
Well yeah I know write a script. I'm having trouble matching the lines and I'm having trouble replacing the columns. I am fairly new at this. I'm thinking awk or sed but I just don't know the right syntax to get it all working.
Ok, but you still haven't given us any details to even BEGIN to try to help you. Just saying "having trouble" gives us nothing to work with.

Again, post samples of the data and what you've tried/written so far, and we'll be glad to assist. We need to see how both sets of input data look now, and what you want when you're done, and what you've written so far.
 
Old 01-31-2013, 12:44 PM   #5
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
Ok here is an example of the data in file1 which has the new file size/date time information that I need in file2

file1:
22696 Jan 31 11:04 2013
23035 Jan 31 10:28 2013
22694 Jan 31 11:04 2013
23033 Jan 31 10:28 2013


etc....etc... repeats many times with different file sizes and times

the other file that i need to merge that data in to looks like this

file2
rwxrwxrwx 0/0 0 Dec 17 15:25 2002 ./
rwxrwxrwx 0/0 0 Dec 17 15:37 2002 ./blah/
rwxrwxr-x 0/3 0 Dec 17 15:37 2002 ./blah/home/
rwxrwxr-x 8654/8655 0 Dec 17 15:53 2002 ./blah/home/foo/
rw-rw-rw- 0/3 889 Jul 29 14:14 2002 ./blah/home/foo/hostlist
rwxrwxrwx 0/3 0 Dec 16 16:02 2002 ./comp0/
rwxrwxrwx 0/3 0 Dec 17 10:23 2002 ./comp0/etc/
rw-rw-rw- 0/3 477 Jul 29 14:14 2002 ./comp0/etc/tab
r--r--r-- 0/3 7329 Jul 29 14:14 2002 ./comp0/etc/bootinfo
rwxrwxrwx 0/3 0 Mar 26 11:07 2003 ./comp0/opt/
rwxrwxrwx 0/3 0 Mar 26 10:45 2003 ./comp0/opt/RARED/
rw-rw-r-- 0/0 133120 Dec 6 16:05 2002 ./comp0/opt/RARED/configs.tar
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/etc/
r--r--r-- 0/0 5797 Dec 10 10:24 2002 ./comp1/etc/bootinfo
r--r--r-- 0/3 22483 Dec 16 15:25 2002 ./comp1/etc/hosts

Again a snippet it continues much longer than this.


I need to replace columns 3-6 with the data from above on the lines that match */etc/hosts. The data above lines up with the lines in the replacement file. Not sure if that helps understand what I'm doing. I really don't know awk or sed but thought they might be of use here somehow.

Last edited by axl718; 01-31-2013 at 03:01 PM.
 
Old 01-31-2013, 12:50 PM   #6
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,101

Rep: Reputation: 288Reputation: 288Reputation: 288
Quote:
Originally Posted by axl718 View Post
Hi,
I have a data file that has many lines in it, that is space delimited. What I want to do is match lines in the file that have the word host in it and then i want to replace columns 3-6 with data that I have in another file. The other file only has the information that belongs on those specific lines. Any ideas?
What happens to the lines in File1 which do not contain the word host?
Are they discarded? Are they written to the output file, unchanged?

What happens if there are more lines in File1 which contain the word host than there are lines in File2?

Daniel B. Martin
 
Old 01-31-2013, 12:56 PM   #7
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by danielbmartin View Post
What happens to the lines in File1 which do not contain the word host?
Are they discarded? Are they written to the output file, unchanged?

What happens if there are more lines in File1 which contain the word host than there are lines in File2?

Daniel B. Martin
The lines in file one that do not contain the word host remain in the file unchanged. There won't be more lines in file1 that contain the word host than there are lines in file2 because file2 was generated after the /etc/hosts file was updated for all the computers in the directory structure which is where file1 was generated from also. However I guess for a sanity check I could count the lines in file1 that match hosts and the lines in file1 and make sure the number is the same.
 
Old 01-31-2013, 12:59 PM   #8
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
I did a sanity check and the number of lines that need to be fixed is 62 and the number of lines that I have data for is also 62.
 
Old 01-31-2013, 01:02 PM   #9
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,101

Rep: Reputation: 288Reputation: 288Reputation: 288
Quote:
Originally Posted by axl718 View Post
I really don't know awk or sed but thought they might be of use here somehow.
You are on the right track. sed and awk are powerful general-use commands. sed is well-suited to text processing; awk is good for that and much more.

It will be helpful if you identify your two input files with names, even if the names are arbitrary such as File1 and File2.

It will be helpful if you give samples of your input files which contain both match and no-match cases.

It will be helpful if you give a sample of the desired output file which corresponds to the sample input files. It gives us a standard of comparison -- if we write a code snippet, we can judge its correctness.

Daniel B. Martin
 
Old 01-31-2013, 01:06 PM   #10
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by danielbmartin View Post
You are on the right track. sed and awk are powerful general-use commands. sed is well-suited to text processing; awk is good for that and much more.

It will be helpful if you identify your two input files with names, even if the names are arbitrary such as File1 and File2.

It will be helpful if you give samples of your input files which contain both match and no-match cases.

It will be helpful if you give a sample of the desired output file which corresponds to the sample input files. It gives us a standard of comparison -- if we write a code snippet, we can judge its correctness.

Daniel B. Martin
I gave samples of the input files above I will edit them and call them file1 and file2 here is an example of what i need the output to be.

rwxrwxrwx 0/0 0 Dec 17 15:25 2002 ./
rwxrwxrwx 0/0 0 Dec 17 15:37 2002 ./blah/
rwxrwxr-x 0/3 0 Dec 17 15:37 2002 ./blah/home/
rwxrwxr-x 8654/8655 0 Dec 17 15:53 2002 ./blah/home/foo/
rw-rw-rw- 0/3 889 Jul 29 14:14 2002 ./blah/home/foo/hostlist
rwxrwxrwx 0/3 0 Dec 16 16:02 2002 ./comp0/
rwxrwxrwx 0/3 0 Dec 17 10:23 2002 ./comp0/etc/
rw-rw-rw- 0/3 477 Jul 29 14:14 2002 ./comp0/etc/tab
r--r--r-- 0/3 7329 Jul 29 14:14 2002 ./comp0/etc/bootinfo
rwxrwxrwx 0/3 0 Mar 26 11:07 2003 ./comp0/opt/
rwxrwxrwx 0/3 0 Mar 26 10:45 2003 ./comp0/opt/RARED/
rw-rw-r-- 0/0 133120 Dec 6 16:05 2002 ./comp0/opt/RARED/configs.tar
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/etc/
r--r--r-- 0/0 5797 Dec 10 10:24 2002 ./comp1/etc/bootinfo
r--r--r-- 0/3 22696 Jan 31 11:04 2013 ./comp1/etc/hosts

Last edited by axl718; 01-31-2013 at 03:09 PM.
 
Old 01-31-2013, 01:58 PM   #11
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
So far I have come up with these commands that will give me the file2 and also find the lines in file1 that need to be changed
ls -Rl | grep hosts | cut -d " " -f 5-8 > file2
sed -n '/hosts/p' file1 > linestobechanged
 
Old 01-31-2013, 02:02 PM   #12
whizje
Member
 
Registered: Sep 2008
Location: The Netherlands
Distribution: Slackware64 current
Posts: 583

Rep: Reputation: 129Reputation: 129
Interesting problem replace each match with next item from list.
 
Old 01-31-2013, 02:39 PM   #13
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,101

Rep: Reputation: 288Reputation: 288Reputation: 288
Quote:
Originally Posted by axl718 View Post
... here is an example of what i need the output to be.
Code:
r--r--r-- 0/3 22696 Jan 31 11:04 2013 ./comp1/etc/hosts
Where did the 2013 come from?
It isn't in either input file.

Daniel B. Martin
 
Old 01-31-2013, 02:45 PM   #14
axl718
LQ Newbie
 
Registered: Apr 2012
Posts: 12

Original Poster
Rep: Reputation: Disabled
It belongs in file1 I forgot to add that column. Sorry
 
Old 01-31-2013, 02:58 PM   #15
danielbmartin
Senior Member
 
Registered: Apr 2010
Location: Apex, NC, USA
Distribution: Ubuntu
Posts: 1,101

Rep: Reputation: 288Reputation: 288Reputation: 288
I changed the input files to make a better test.
File1 ...
Code:
22696 Jan 31 11:04 2013
23035 Feb 01 10:28 2013
22694 Feb 03 11:04 2013
23033 Feb 05 10:28 2013
File2 ...
Code:
rwxrwxrwx 0/0 0 Dec 17 15:25 2002 ./
rwxrwxrwx 0/0 0 Dec 17 15:37 2002 ./blah/
rwxrwxr-x 0/3 0 Dec 17 15:37 2002 ./1b_2a_2b_4a_4b_a11_a21_c11_c21_cndadj/home/
rwxrwxr-x 8654/8655 0 Dec 17 15:53 2002 ./blah/home/foo/
rw-rw-rw- 0/3 889 Jul 29 14:14 2002 ./blah/home/foo/hostlist
rwxrwxrwx 0/3 0 Dec 16 16:02 2002 ./comp0/
rwxrwxrwx 0/3 0 Dec 17 10:23 2002 ./comp0/etc/
rw-rw-rw- 0/3 477 Jul 29 14:14 2002 ./comp0/etc/tab
r--r--r-- 0/3 7329 Jul 29 14:14 2002 ./comp0/etc/bootinfo
rwxrwxrwx 0/3 0 Mar 26 11:07 2003 ./comp0/opt/
rwxrwxrwx 0/3 0 Mar 26 10:45 2003 ./comp0/opt/RARED/
rw-rw-r-- 0/0 133120 Dec 6 16:05 2002 ./comp0/opt/RARED/configs.tar
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/etc/
r--r--r-- 0/0 5797 Dec 10 10:24 2002 ./comp1/etc/bootinfo
r--r--r-- 0/3 22483 Dec 16 15:25 2002 ./comp1/etc/hosts
r--r--r-- 0/3 22483 Dec 17 15:35 2002 ./comp1/etc/hosts
r--r--r-- 0/3 22483 Dec 18 15:45 2002 ./comp1/etc/hosts
r--r--r-- 0/3 22483 Dec 19 15:55 2002 ./comp1/etc/hosts
This code ...
Code:
awk 'BEGIN {while (getline < "'"$InFile1"'") a[++j]=$0};
     {if ($0 !~ /hosts/) {print}
   else
     {print $1,$2,a[++k],$8} }' $InFile2 > $OutFile
... produced this output file.
Code:
rwxrwxrwx 0/0 0 Dec 17 15:25 2002 ./
rwxrwxrwx 0/0 0 Dec 17 15:37 2002 ./blah/
rwxrwxr-x 0/3 0 Dec 17 15:37 2002 ./1b_2a_2b_4a_4b_a11_a21_c11_c21_cndadj/home/
rwxrwxr-x 8654/8655 0 Dec 17 15:53 2002 ./blah/home/foo/
rw-rw-rw- 0/3 889 Jul 29 14:14 2002 ./blah/home/foo/hostlist
rwxrwxrwx 0/3 0 Dec 16 16:02 2002 ./comp0/
rwxrwxrwx 0/3 0 Dec 17 10:23 2002 ./comp0/etc/
rw-rw-rw- 0/3 477 Jul 29 14:14 2002 ./comp0/etc/tab
r--r--r-- 0/3 7329 Jul 29 14:14 2002 ./comp0/etc/bootinfo
rwxrwxrwx 0/3 0 Mar 26 11:07 2003 ./comp0/opt/
rwxrwxrwx 0/3 0 Mar 26 10:45 2003 ./comp0/opt/RARED/
rw-rw-r-- 0/0 133120 Dec 6 16:05 2002 ./comp0/opt/RARED/configs.tar
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/
rwxrwxrwx 0/0 0 Dec 16 17:01 2002 ./comp1/etc/
r--r--r-- 0/0 5797 Dec 10 10:24 2002 ./comp1/etc/bootinfo
r--r--r-- 0/3 22696 Jan 31 11:04 2013 ./comp1/etc/hosts
r--r--r-- 0/3 23035 Feb 01 10:28 2013 ./comp1/etc/hosts
r--r--r-- 0/3 22694 Feb 03 11:04 2013 ./comp1/etc/hosts
r--r--r-- 0/3 23033 Feb 05 10:28 2013 ./comp1/etc/hosts
Daniel B. Martin
 
2 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] delete a matched line from the file+shell script novicunix Programming 13 02-01-2013 08:37 AM
[SOLVED] Replace pattern in specific lines and column with AWK cgcamal Programming 10 04-26-2010 01:11 AM
Text file manipulation: selecting specific lines/columns using awk and print CHARL0TTE Linux - Newbie 2 02-27-2010 02:40 AM
Comparing two files to get matched contents in another file using shell script pooppp Linux - Networking 3 08-05-2008 12:11 AM
How can I use a shell script to add and replace lines in a file? abefroman Programming 10 12-27-2005 05:05 PM


All times are GMT -5. The time now is 01:56 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration