ssh using infiband only

mvairavan · 09-02-2012, 04:42 PM

Hi i have a server which just has infiniband. When i try to ssh using the infiniband ip it doesn't work. But it can ping the ip.

host node - has both ethernet and infiniband
remote node - only infiband.

How do i do this?
Thank you.

mvairavan · 09-02-2012, 06:22 PM

Ok here is the whole story. This may make the problem more easier to understand.

Host machine - is where i am logged in
Remote machine - One i am trying to connect.

Case 1: Both the machines have both ethernet and infiband plugged in.
ssh from host machine to remote machine's infiband ip , worked fine i am logged in.

Now while i am logged in i plugged out the ethernet from remote machine,
(NOTE: I am logged in using IB ip not ethernet ip)
I am still logged in for a few secs may 40-60s, i can do all commands it works. (So surely this is through infiniband)
But after 60s the connection is lost but it hasn't logged me out i am just struck there without anything working.

So at this point i went back and plugged in my ethernet cable back on remote machine. I get back the connection as i was before!!

So whats happening here??

Case 2: Host machine has both ethernet and infiniband, remote machine has only infiniband.
ssh from host to remote's IB ip. It gets struck before server accepts the key.
ssh -v output is as follows

Code:

[<user>@host ~]$ ssh -v 10.2.255.127
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 10.2.255.127 [10.2.255.127] port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/identity type -1
debug1: identity file /home/<user>/.ssh/id_rsa type 1
debug1: identity file /home/<user>/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '10.2.255.127' is known and matches the RSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:73
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/<user>/.ssh/identity
debug1: Offering public key: /home/<user>/.ssh/id_rsa

It just waits there...
So now i go and plug in my ethernet cable, then i got logged in..!!!
Hufff!!

Wats happening here and how could i solve this?
Thanks in advance?

NOTE:Ping to the IB ip works fine anytime

btmiller · 09-02-2012, 07:38 PM

That's odd ... it looks like SSH is making the initial connection but hanging on authentication. Sometimes SSH will hang for about a minute trying to do a DNS lookup on the incoming connection. You might try setting "UseDNS no" in the sshd_config on the remote machine and see if you can connect then.

The other thing to check would be thew routing tables between the two machines. That is you want to make sure that your host is actually using the IB interface to actually communicate with the remote machine.

mvairavan · 09-04-2012, 03:49 PM

It is already "USE DNS no " in sshd config and i am directly specifying the ip anyway.

But to the second part, routing tables. I dint understand what you are asking me to check, can you explain a bit more.
I have assigned IP to these IB ports just by putting ifcfg-ib0 file inside network-scripts. These two machines are conenct via a IB switch.

btmiller · 09-04-2012, 09:12 PM

To check the routing tables, run "netstat -rn" on both machines and post the results.

mvairavan · 09-05-2012, 10:57 AM

Code:

[<user>@host ~]$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
255.255.255.255 0.0.0.0         255.255.255.255 UH        0 0          0 eth0
<Public IP>     10.1.1.1        255.255.255.255 UGH       0 0          0 eth0
10.2.0.0        0.0.0.0         255.255.0.0     U         0 0          0 ib0
10.1.0.0        0.0.0.0         255.255.0.0     U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ib0
224.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth0
0.0.0.0         10.1.1.1        0.0.0.0         UG        0 0          0 eth0
[<user>@<host> ~]$ ssh <remote> "netstat -rn"
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
255.255.255.255 0.0.0.0         255.255.255.255 UH        0 0          0 eth0
<Public IP>     10.1.1.1        255.255.255.255 UGH       0 0          0 eth0
10.2.0.0        0.0.0.0         255.255.0.0     U         0 0          0 ib0
10.1.0.0        0.0.0.0         255.255.0.0     U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ib0
224.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 eth0
0.0.0.0         10.1.1.1        0.0.0.0         UG        0 0          0 eth0

Hope this helps.