Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
In other words check abc in File1 against File2, is it exists in File2 then skip, otherwise append it to the file, then check def, then tex etc etc
You don't specify, so I'll infer from your example that the keys are in ascending order in both input files. Then this will write the desired result to stdout:
Code:
sort -t: -msuk1,1 File2 File1
man sort.
To update File2, don't directly redirect output, because that would truncate File2 before sort reads it. Rather,
Code:
sort -t: -msuk1,1 File2 File1 > tmp
mv tmp File2
More generally, I think your objective is to merge two "associative arrays" (AKA "dictionaries" in PostScript or "hashes" in Perl). Do you care to tell us why you want to do this?
More robustly and flexibly than the above "sort" call, you could use Perl to build up the hash %t, then write it all out at the end. Note the opposite order of arguments:
Code:
perl -we 'while(<>) {($k,$v)=split /:/, $_, 2; $t{$k}=$v;} while (@e = each %t) {print join ":", @e}' File1 File2
Or, if you want the output rows ordered (and with nicer names):
I've played around with awk and for loops but don't seem to be getting anywhere
I learned csh scripting, and awk, and sed, and (ba)sh. When I came across Perl, I realized that was the one tool I should have leared in the beginning. Give it a try!
Quote:
Originally Posted by kapilsingh
uniq file1 file2
That keeps the unique lines from file1 (only!) and overwrites file2. Not the solution. Also, as gregmcc points out, this looks at the whole line.
You can tell uniq to only heed the first 3 characters on each line. That could do in the example, but it's not really colon-delimited.
/Christian
Last edited by Quigi; 04-22-2009 at 02:27 PM.
Reason: Add "to", required by English grammar
I should have specified - The keys are in a random order
Quote:
Originally Posted by Quigi
You don't specify, so I'll infer from your example that the keys are in ascending order in both input files. Then this will write the desired result to stdout:
Code:
sort -t: -msuk1,1 File2 File1
man sort.
I tried this and it works great if the files are already sorted.
Quote:
More generally, I think your objective is to merge two "associative arrays" (AKA "dictionaries" in PostScript or "hashes" in Perl). Do you care to tell us why you want to do this?
File1 is on one server and File2 is on another server. I want to keep File2 up to date with new info that is added to File1. But I couldnt do a copy or rsync as the file content is not exactly the same.
I ended up using this:
Code:
awk -F ":" 'BEGIN{while(getline<"/tmp/file1") a [$1]=1 } ; a [$1] !=1 {print $0 } ' /tmp/file2 > /tmp/file.diff
I should have specified - The keys are in a random order
I tried this and it works great if the files are already sorted.
As you probably saw in the man page, "-m" tells sort that the files are already sorted. If they aren't, simply drop the "m", and sort will order them. The keys will be in order in the output. I can't tell from your example if that's a problem.
Or use one of the Perl one-liners.
Quote:
File1 is on one server and File2 is on another server. I want to keep File2 up to date with new info that is added to File1. But I couldn't do a copy or rsync as the file content is not exactly the same.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.