LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 09-01-2010, 07:12 AM   #1
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Rep: Reputation: 0
Question add second field of two files on different servers


Hi

I have one file placed at one server whose sample as follows:
57272 28
56767 0
57575 4
58888 6
53030 26
54242 0
56060 0

And another file at different server whose sample data as follows:
57272 22
56767 0
57575 1
58888 2
53030 13
54242 0
56060 1

Now i want to run a script from one of the server which adds second field from both files which are not on same server and gives the output as follows:
57272 50
56767 0
57575 5
58888 8
53030 39
54242 0
56060 1
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-01-2010, 07:27 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

There's not enough info to help.....

Do these 2 files always have the same first columns (same number in the same location) or is the following also possible???

File one:
57272 28
57575 4
58888 6
53030 26

File two:
57272 22
57575 1
58888 2
53030 13
56060 1

Please elaborate a bit.
 
Old 09-01-2010, 08:37 AM   #3
Chirel
Member
 
Registered: Nov 2009
Posts: 55

Rep: Reputation: 19
Hi,

Let's say you can connect on both server using ssh and using public/private keys.
Let's say also that :

File 1 is /app1/f-one on server1.foo.org
File 2 is /app2/f-two on server2.foo.org

Then first you could create a script doit.sh :
Code:
#!/bin/bash
declare -a akey
declare -a aval

idx=0
while read k v; do
    [ -z ${aval[$k]} ] && akey[$((idx++))]=$k
    aval[$k]=$((${aval[$k]:-0}+$v))
done

i=0
while [ $i -lt $idx ]; do
    echo "${akey[$i]} ${aval[${akey[$((i++))]}]}"
done
It's obvious that you will not forget to chmod +x doit.sh, then all you have to do is :

Code:
# { ssh server1.foo.org "cat /app1/f-one"; ssh server2.foo.org "cat /app2/f-two"; } | ./doit.sh

Last edited by Chirel; 09-01-2010 at 08:40 AM.
 
2 members found this post helpful.
Old 09-01-2010, 08:41 AM   #4
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

@Chirel if the two files are always the same (saurabhmehan still hasn't answered that one), it can be done a bit simpler:
Code:
#!/bin/bash

# machines to fetch file from
serverOne="1.0.0.1"
serverTwo="1.0.0.2"

# user to use on remote machines
userOne="user1"
userTwo="user2"

# fetch the files (passwordless scp is assumed)
scp $userOne@$serverOne:path/to/fileone /tmp/tmp.$serverOne
scp $userTwo@$serverTwo:path/to/filetwo /tmp/tmp.$serverTwo

# create new file
diff -y /tmp/tmp.$serverOne /tmp/tmp.$serverTwo | \
awk '{ print $1, $2+$5 }' > /path/to/outfile

exit 0
 
2 members found this post helpful.
Old 09-01-2010, 08:49 AM   #5
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,087

Rep: Reputation: 634Reputation: 634Reputation: 634Reputation: 634Reputation: 634Reputation: 634
assuming nfs or sshfs:
Code:
[schneidz@hyper temp]$ paste /mnt/sshfs-server1/path/to/input1 /mnt/sshfs-server2/path/to/input2 | awk '{a = $2 + $4}  {print $1 " " a}'
57272 50
56767 0
57575 5
58888 8
53030 39
54242 0
56060 1
 
Old 09-01-2010, 08:52 AM   #6
Chirel
Member
 
Registered: Nov 2009
Posts: 55

Rep: Reputation: 19
Hi,

@druuna, you are right

But as i did not know i consider the "hard" way

I must admit i love the awk + diff way
 
Old 09-01-2010, 09:00 AM   #7
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,

@Chirel: One tends to use the tools one knows. The longer you work with unix/linux the more tools you come across (and sometimes forget again...).

Your solution has one advantage over mine: It also works when the infiles are not the same.....
A simple sort might take care of that problem in my script, but that might not be what the OP wants.

Last edited by druuna; 09-01-2010 at 09:02 AM.
 
Old 09-03-2010, 12:28 AM   #8
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Smile add second field of two files on different servers

Hi,

@drunna thanks for your post, i want to tell you the file structure is same but count will be different.

But i want to do it without copying the file from both servers.

Quote:
Originally Posted by druuna View Post
Hi,

@Chirel: One tends to use the tools one knows. The longer you work with unix/linux the more tools you come across (and sometimes forget again...).

Your solution has one advantage over mine: It also works when the infiles are not the same.....
A simple sort might take care of that problem in my script, but that might not be what the OP wants.
 
Old 09-03-2010, 12:37 AM   #9
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Question add second field of two files on different servers

Hi,

@schneidz: Please provide me the sample command with the dummy inputs as i am trying following command and its not working:
Code:
paste /mnt/sshfs-10.0.0.1/home/smehan/applog/cp_report.txt /mnt/sshfs-10.0.0.2/home/smehan/applog/cp_report.txt | awk '{a = $2 + $4}  {print $1 " " a}'



Quote:
Originally Posted by schneidz View Post
assuming nfs or sshfs:
Code:
[schneidz@hyper temp]$ paste /mnt/sshfs-server1/path/to/input1 /mnt/sshfs-server2/path/to/input2 | awk '{a = $2 + $4}  {print $1 " " a}'
57272 50
56767 0
57575 5
58888 8
53030 39
54242 0
56060 1
 
Old 09-03-2010, 12:50 AM   #10
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Question add second field of two files on different servers

Hi,

@drunna: Yes the first column will be same in both files of different servers that is same number in same location and below strategy which u mention according to my case not possible.

Quote:
Originally Posted by druuna View Post
Hi,

There's not enough info to help.....

Do these 2 files always have the same first columns (same number in the same location) or is the following also possible???

File one:
57272 28
57575 4
58888 6
53030 26

File two:
57272 22
57575 1
58888 2
53030 13
56060 1

Please elaborate a bit.
 
Old 09-03-2010, 01:14 AM   #11
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
assuming you already set up passwordless ssh.

Code:
ssh root@localhost "cat /path/file1" | awk 'BEGIN{
  while( ( getline line<"/path/file2") > 0 ){
   m=split(line,a," ")
   array[a[1]]=a[2]
  }
}
{
   array[$1]+=$2
}
END{
  for(i in array) print i ,array[i]
}'
 
Old 09-03-2010, 01:49 AM   #12
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374Reputation: 2374
Hi,
Quote:
Originally Posted by saurabhmehan View Post
But i want to do it without copying the file from both servers.
Have a look at Chirel's or ghostdog74's solutions.
 
Old 09-03-2010, 01:58 AM   #13
saurabhmehan
Member
 
Registered: Jul 2010
Posts: 44

Original Poster
Rep: Reputation: 0
Smile add second field of two files on different servers

Hi Guys,

@druuna,@Chirel,@ghostdog74 thanks for providing me hints to solve my problem.
I have done with it with one liner.Check the below code:
Code:
ssh hostname "cat serverfile" | awk 'NR==FNR{_[$1]=$2; next} {print $1, _[$1]+$2}' localfile -
Quote:
Originally Posted by druuna View Post
Hi,

Have a look at Chirel's or ghostdog74's solutions.
 
Old 09-03-2010, 08:53 AM   #14
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,087

Rep: Reputation: 634Reputation: 634Reputation: 634Reputation: 634Reputation: 634Reputation: 634
Quote:
Originally Posted by saurabhmehan View Post
Hi,

@schneidz: Please provide me the sample command with the dummy inputs as i am trying following command and its not working:
Code:
paste /mnt/sshfs-10.0.0.1/home/smehan/applog/cp_report.txt /mnt/sshfs-10.0.0.2/home/smehan/applog/cp_report.txt | awk '{a = $2 + $4}  {print $1 " " a}'
my first post does have the sample command with the dummy inputs. if yours is not working i suspect your server isnt mounted in that directory (notice that my original post said: assuming nfs or sshfs). you can test that by posting the result of:
ls /mnt/sshfs-10.0.0.1/home/smehan/applog/cp_report.txt /mnt/sshfs-10.0.0.2/home/smehan/applog/cp_report.txt

Last edited by schneidz; 09-03-2010 at 08:57 AM.
 
  


Reply

Tags
array, asap, bash, calc


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
can i add my own field(of 2 bytes) in the ip header structure raklo Linux - Networking 5 08-18-2011 02:17 PM
[SOLVED] merge 2 files with AWK by the field value dayamoon Linux - Newbie 8 06-03-2010 02:06 AM
Shell regex - add new field czezz Programming 3 02-28-2009 03:13 PM
Perl add numbers in 2nd field twantrd Programming 6 10-19-2006 08:26 PM
How to extract the 16 byte source name field (Name to add) from the NetBIOS fram (NBF Bassam Linux - Networking 0 05-23-2004 04:58 AM


All times are GMT -5. The time now is 04:27 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration