LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-27-2011, 08:06 AM   #1
silkysue
LQ Newbie
 
Registered: Jan 2011
Posts: 2

Rep: Reputation: 0
awk command to merge two files


Hi all

Hoping for some help with a script I need to implement. New-ish to shell scripting but no-one else here has any clue about shell scripting, let alone AWK and I am struggling.

I have 2 files, the 1st column in both is the RecID so to speak. The Merge file will contain all records contained in Prim file. In addition any records which aren't in the primary file but in the SecFile will get added to the Merge File.

PrimFile:
DE3001 16/06/09 P
DE4001 16/06/09 P
DE4101 16/06/09 P
DU3101 16/06/09 0

SecFile:
DE4101 13/06/04 0
DU3101 27/01/11 P
DS2111 19/01/07 P

MergeFile:
DE3001 16/06/09 P
DE4001 16/06/09 P
DE4101 16/06/09 P
DU3101 16/06/09 0
DS2111 19/01/07 P

Appreciate any help..

Suzanne
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 01-27-2011, 08:13 AM   #2
onebuck
Moderator
 
Registered: Jan 2005
Location: Midwest USA, Central Illinois
Distribution: SlackwareŽ
Posts: 12,603
Blog Entries: 25

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Hi,

Welcome to LQ!

What have you done to find a solution to the problem? Other than to post here.

We will aid you when you help yourself to a solution. Provide us with what you have attempted and then maybe someone will be able to assist. 




Just a few links to aid you to gaining some understanding. Sure some may seem beyond a newbie but you must start somewhere;



Linux Documentation Project
Rute Tutorial & Exposition
Linux Command Guide
Utimate Linux Newbie Guide
LinuxSelfHelp
Bash Beginners Guide
Bash Reference Manual
Advanced Bash-Scripting Guide
Linux Home Networking



The above links and others can be found at 'Slackware-Links'. More than just SlackwareŽ links!
 
Old 01-27-2011, 10:18 AM   #3
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,051

Rep: Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852
something like:
Code:
for id in sec-file
 if [ `grep $id` not in prim-file ]
 then
  echo $row > to prim-file
 fi
done
would probably work.

Last edited by schneidz; 01-27-2011 at 10:21 AM.
 
Old 01-27-2011, 10:44 AM   #4
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
Hi,

Code:
#!/bin/bash

awk 'BEGIN { while ( ( getline < "PrimFile" ) > 0 )
uniquarray[$1] = $2" "$3
}
{
if ( ! uniquarray[$1] ) {
  uniquarray[$1] = $2" "$3 } 
}
END {
for (i in uniquarray) {print i,uniquarray[i]}
}' SecFile | sort > MergeFile

exit 0
As stated by onebuck, you should post what you have tried or what parts are unclear. On the other hand I do believe that an awk solution for this problem isn't something that can be easily done by someone without any scripting knowledge......

I won't include a description of how the above script works, that's for you (silkysue) to find out (and hopefully learn something in the process).

Anyway, hope this helps.
 
2 members found this post helpful.
Old 01-27-2011, 10:53 AM   #5
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,424

Rep: Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823
Maybe something like:
Code:
awk 'FNR==NR{arr[$1]++;print;next}!($1 in arr)' PrimFile SecFile > MergeFile
 
Old 01-27-2011, 11:02 AM   #6
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
@grail: Nice and short!!
 
Old 01-27-2011, 11:08 AM   #7
silkysue
LQ Newbie
 
Registered: Jan 2011
Posts: 2

Original Poster
Rep: Reputation: 0
Hi guys

Thanks for all the responses, lots of googling and even a little bit of learning has got me there!

Quote:
awk '{a[$1]=$0}END{for(i in a)print a[i]}' oldfile primaryfile | sort > mergefile
So Druuna your solution also worked and am looking through it trying to see how it works:
1st field in PrimFile is the key, stick into an array
Loop through the array and for any key in SecFile but not PrimFile print to file
Sort to Mergefile

Thanks all!!

SOLVED
 
Old 01-27-2011, 11:14 AM   #8
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
Hi,

You're welcome

One small comment:

Quote:
Originally Posted by silkysue View Post
So Druuna your solution also worked and am looking through it trying to see how it works:
1st field in PrimFile is the key, stick into an array
Loop through the array and for any key in SecFile but not PrimFile print to file
Sort to Mergefile
You have the bold part backwards....

SecFile is "looped" (read line by line by awk) and its first field is checked against the array. If it doesn't exist; add it.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] merge 2 files with AWK by the field value dayamoon Linux - Newbie 8 06-03-2010 03:06 AM
awk merge and sum lines problem lalo4080 Programming 4 08-12-2008 11:21 AM
shell command using awk fields inside awk one71 Programming 6 06-26-2008 05:11 PM
awk command to merge columns from two separate files into single file? johnpaulodonnell Linux - Newbie 4 01-23-2007 11:10 AM
Is there a command to merge two files as two columns of one file? davee Linux - General 2 07-19-2005 11:52 AM


All times are GMT -5. The time now is 08:04 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration