Hi,

I have a very weird problem that's driving me nuts. The situation is as follows:

A company with an office in City A and an Office in City B, connected via VPN. I am sitting at A and want to ssh to a server at B. The connection is as follows:

me -> router A -> VPN Appliance A -> (internet) -> VPN Appliance B -> server

However, it's slightly different from the view of the server (there's another router):

server -> router B -> VPN appliance B -> (internet) -> VPN appliance A -> router A -> me

The server, let's call it 'virtual', can ssh to me perfectly. But when I try to ssh to virtual, nothing happens on my side - no errors, no timeout, no greeting/login, just nothing. There is a message in the server log file: "Did not receive identification string from UNKNOWN", it appears everytime I try to connect.

A solution to fix this is to add a route to the network of City A with the VPN appliance as gateway to the server 'virtual'. This way, it won't try to go via its default gateway (router B) and from there to the VPN appliance, but to go to the appliance directly, like this:

server -> VPN appliance B -> (internet) -> VPN appliance A -> router A -> me

However, I don't understand why this is necessary, because that defeats the whole purpose of a router, and apart from this issue, the routing seems to work perfectly (remember, SSH'ing from the server to me always works, regardless of having the extra route set on the server or not). Also, traces and pings work in both directions with and without the route. Its just that somehow ssh doesn't work in one direction if the route is not set.

To make things even more complicated, there are two more servers next to the server I am having issues with, 'sun' and 'web'. They are connected to the same switch and are in the same network as 'virtual'. I can ssh to these servers without problems - and they do NOT have any extra routes set. So I think, whatever the problem is must have something to do with the server 'virtual', but not with the router B or the rest of the chain.

What I did was to SSH to one of the "working" servers, 'sun', and from there to 'virtual', until I found out adding the route would solve the problem.

This worked for some time. I noticed that my ssh connection to 'sun' would drop sometimes (connection reset by peer) after 10-30 minutes or so. Also, sometimes I needed to try twice before I was presented the password dialog. I didn't pay much attention to that though, as my ssh problem with virtual was more important.
Then, starting today, I couldn't ssh to 'sun' anymore, with exactly the same symptomatics like I had it with virtual. No timeout, no error, no nothing. The log entry there was slightly different: "Did not receive identification string from 10.0.10.151". So where virtual said "unknown", 'sun' at least knows my IP. I already knew what to do, so I added the route to the VPN appliance on 'sun' and it was working again. As far as I can tell, without any interruptions or connection drops like I had before.

Then there is the 'web' server. I didn't add the route there yet, and yet I have no issues at all SSH'ing to it. But who knows, maybe in a few more days it might start behaving like virtual and sun.
(as I write this, I wanted to check for the debian version there, and on pressing a key in its open SSH window, the connection was reset by peer...)

Now one could say "so what, just add the route to every server you have there and it's all solved", and this might even be correct. It's just that I don't understand this *at all*. The router is there, it has all important routes and it seems to be routing correctly - I mean, after all, I can still SSH to the 'web' server and all three servers can SSH to me (even without the local static routes being set).

I am afraid adding the route to every server which stops accepting SSH connections would just be a dirty workaround but it wouldn't fix the problem. That's why I'm here and tell you guys this story.

Sidemarks: sun runs debian 6, virtual runs CentOS 5.6, web runs debian 4.

So, does anyone of you have an idea what could be wrong here? If you need any more information, debug output, traces etc... please let me know. I'm at my wits' end.

Thank you.

Regards, Jay