Howdy,

i'm testing a solution using ssh. i'm running tcpdump to see what's going on and the first time i saw ssh send one tcp packet, pause, timeout, send another etc... the second time i tried it it sent two packets, pause, timeout, send another two. this has gone on very reliably and i just tried it for the 8th time and sure enough i see 8 packets going out to that address.

wtf is going on here? what part of the client is making ssh try harder each successive time to reach the destition server? surely it's none of the actual ssh clients business to know who it's previously tried to reach? is it netfilter??? i'm both intrigued and clueless.... anyone?

Cheers

Chris

That sounds very odd. Maybe that "solution" you are testing is bugged some how and is making ssh client to misbehave ?

[logical question - sorry if I'm offending you :P]
Or do you have multiple ssh clients running ?
If you use some script and it starts a new client and ...

Cheers

/////

no no not at all. just the bog standard ssh client directly.

I wanted to reply earlier but since I don't have The Answer I didn't want to be the one to take this post off the 0 reply list. When I read this I decided that it was time that I became more familiar with how ssh works so I went to the source, FreeBSD. They had a nice tutorial on setting it up but what I wanted was information on how it is designed so that I could answer your question about one packet tries followed by two packet tries. I didn't find any information about how ssh is designed, at least not at that level of detail. Second, when it comes to diagnosing this failure I would use the following diagnostic techniques to investigate the problem hoping that I would find lower level details about what exactly is happening. This requires root access on the target machine.

You said that you have run tcpdump on the client to see this behavior. If you have access to root on the target machine I would run tcpdump there to see what that machine detects. Since tcpdump reads net traffic before the firewall gets to it you can see what the raw NIC sees. This may seem like it would just be what you see on the client but what if the target doesn't see it? That would be helpful diagnosing the problem. (If the packets are not getting past a router for instance.)

Look on the target machine's syslog logs. Look for security violations. I have found that access problems are very very often related to security. The firewall may be silently dropping your packets. The target machine would have details about whether this is happening, and if it is, then why it is happening.

Another security possibility is if the two machines have different ssh keys when they should have a common key.

It is possible that the target machine has a list of approved ssh accounts and the account on the client is not on this list. This is particularly possible if the target machine does not allow remote logins for root, if you are using root on the client. But any user account can be blacklisted if ssh has a list of approved users. It's a lot like the cron.allow file. If it has any entries then cron only allows accounts named in that file to use cron.

That's where I would start. Target machine: security problems.

well to be honest the destination ip address does not exist (we were testing as part of a plan to snat that address to a real server) so i know that there is nothing on the server side at all to complicate things. Obviously in line with this i get no reply either. it just seems that something on my client knows that the previous attempts failed, it's really bizarre.

Have you tried completely closing out your client, e.g. closing and making sure it's not in memory, and then trying again? If you don't get duplicate retries as you stated earlier then it definetly is your client and you probably want to look into this. If not then it has to be something else.

The IP that you are sshing to, is that on the same broadcast domain? or is it being routed? Might be some wierd router -> ssh communication if it's being routed. (Which can happen. Had a router go down before because i used ssh keys to auth. Some reason the router saw it and decided to initiate a vpn tunnel request and then choked on it. True story and that bug has since been fixed on that particular vendors router.)

the destination subnet is in another data centre in another country. but again i never see a single response back from that host, or rather reporting to be from that host if other things would get in the way (like our load balancers and such like can do, when they spoof TCP RST replies).

my client is just CLI ssh, so it's totally unloaded after the event, i'm not that knowledgable about ram cache... maybe the client is reclaiming the same memory space it used previously? seems the only thing i could stab at to make it at all tangible.