Peculiar behavior of ssh: hangups, changing host key
 

I installed Centos 5.5 on a new computer (SuperMicro H8DGU) yesterday. Some odd things happen sporadically when I connect to it by ssh from a terminal emulator. Mostly I use Van Dyke's SecureCRT on a Windows machine, but not exclusively.

1. Occasionally the connection drops, and when I log in again I'm told "The host key sent by the server is different from the host key stored in the host key database." Then it gives me the MD5 hash of the host key fingerprint. The odd thing is that this hash alternates between two different values! Just two!

I just ran ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key.pub, and I recognize the output as one of the two hash values. The date on the file is yesterday afternoon, so that hasn't changed.

2. Sometimes there is an error message "Auth User/Pass with PS...fail...Please reconnect!." This is preceded by an "unspecified GSS error", if I remember correctly.

I'm using password authentication.

I can't find any relevant error messages in /var/log/secure, just "password accepted" and "end session" lines.

I have compared this machine to a similar machine (Scientific Linux 5.5) that works properly. The sshd_config files are identical. So are /etc/pam.d/sshd and /etc/pam.d/system_auth. In fact I haven't messed with anything in the sshd configuration.

How have you assigned IP for that system, static or via DHCP?

Also, is it possible that you have another system with the same IP (maybe secondary IP so you missed it)?

The IP is static.

I'm not sure about the possibility of another system with the same IP. The number was assigned by our network manager. Usually there's some sort of error message when there's a conflict like that, but I haven't seen anything.

While dredging through the logs, I found some error messages from avahi-daemon. The machine that works properly is not using it. I don't think we use it for anything. Could this be related to my problem? I shut it off, so maybe I'll get an answer in a few hours.

Maybe some PC do use the same IP, who knows why. When you see different ssh key disconnect that PC from network and try pinging his IP.

You can also take a look at hostname of the "second ssh key" system, and look for files that are missing or should not be there (wherever you have access to files). You can also check for MAC address of the NIC and compare them (this can help if there is really another system with the same IP to track it down.)

No luck so far. I disconnected the cable and pinged the address. Nothing answered.

Turning off avahi-daemon didn't help.

In my known-hosts file, the hostname of the "second ssh key" system is always the same. It's the one the DNS server gives for the IP address. I can't figure out a way to use the DSA signature to find particular machine.

You pinged it from different system right? Just checking.
I never said DSA signature or mentioned DNS server.

I said when you are warned that ssh-key has changed for the host you are trying to log in accept and log in. Then look around for hostname on that system you are logged in and try to get MAC address. You will need root privileges for this, or use some service that will show your MAC to some service you have on your network. That was my suggestion, to see if you are logging to different PC by any chance.

Thanks. Now I understand what you meant.

Unfortunately I haven't been able to log in to the "extra" host. That must be the machine that give me the "Auth User/Pass with PS...fail...Please reconnect!." messages. So far its identity remains a mystery.

At the suggestion of our network manager I switched the IP addresses and names of the new machine with an old one that works. It really begins to look like there's an extra machine using the new IP. The old machine with the new IP refused to start eth0 at boot time, saying that there's another machine using the address. That's pretty definitive. When I try to ssh to it, I get the "Auth User/Pass...." error, which it couldn't give with eth0 down.

The new machine with the old IP hasn't given any trouble yet, but it's only been a few minutes.

I will throw this to our network manager. Maybe his router logs will show the MAC of the machines that use the IP. Or maybe his records will show who had the IP before this week.

Now that your ssh server system has different IP, any PC/router can sy MAC of the culprit system. should give you it's MAC address.

My part of the problem is solved. The network manager assigned a new IP address which has no interference from other machines. Using arp as you suggested, I found the hardware address and passed it on to him. Now it's his problem to track it down.