I have two clients set up for public key auth only to a server...tonight, both clients indicated that the id/key of the server had changed.

I checked the mod dates for the key files in /etc/ssh on the server and they haven't changed...

I did change the groups that the login account is a member of just before I started getting the notice on the two clients...could this change possibly cause the client to report a change in the sshd id?

I'm running fail2ban on sshd and have password login and root login disabled...watching the secure log carefully, I pretty certain (ie. 99.999%) nobody has logged in via ssh. What else would possibly be causing this?

Yikes. You're saying your ssh client reported that the sshd server's host keys changed, right?

Nope. IMO, the timing is a coincidence.

Is there anything else you've changed recently on the server side? (Was the server's hostname or IP previously assigned to a different host at some point?)

---

BTW, what distro / version? (And what sshd version?)

Yes, two separate clients are both reporting the server host key changed...

The server is OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-fhel5

Server is running CentOS 5.3...

I would definitely not take the warning lightly, unless you can explain the change. (i.e. This protection is in place for a good reason.)

Is it possible the contents of your clients' ~/.ssh/known_hosts file could have become altered / corrupted somehow?

Yeah, my thoughts as well...Given that both clients reported the problem within an hour of each other, I have to pin it on the server changing and not a client corruption or the like. But, both clients are on the same dhcp lan, so maybe somehow that could affect things?

I scanned /var/log/secure, /var/log/messages and the apache logs and didn't see anything odd...rkhunter doesn't detect anything.

Got a backup copy of the server keys? Try a comparison.

There has to be a satisfactory explanation here. Try to think hard about any other recent changes at the server level. Are you the only sysadmin? Is this a 'net-facing host? Do you monitor system activity regularly?

You didn't answer about any hostname / IP trading among servers, so I'm assuming that is not the case.

It would be nice if you were running a HIDS so that you could verify host key SHA256 / MD5 sums. Mtime alone is not enough to verify file integrity. [ edit: Or a known good backup would help, as mentioned above. ]

I'll reiterate what I said earlier: until you can explain the problem, the server you're logging in to (possibly MITM) is suspect and should be treated with caution.

Yes, I am the only admin. This is a new replacement system (net facing) which has the same host name as the previous (plus new ip)...I did change the hostname from its original when it was brought up, but that happened about a week before this event, and I had been logging in fine with the new (changed) hostname.

Unfortunately, I don't have backups of /etc at this point to check the keys...

We haven't identified the root cause here. These are the options, IMO:

• Bite the bullet and rebuild the system, so that you're starting with a known clean slate. (I'm leaning toward "this host has not been cracked, and there just hasn't been enough investigation"...)
• Physically sit in front of the server and log in to a tty. Then log in over ssh and confirm that you see those remote connections from the physical server. (This would help alleviate the main concern, namely another host posing as you.) Once confirmed, remove the known_hosts entries from your ssh clients and go on with life.

In the future please take regular backups and consider running a HIDS.