Weird DNS Issue

I recently migrated our DNS/LDAP/NFS server to a new machine. The Old machine has taken the IP address of what now is the New machine. IE:
Old Machine (Before Switch): orgo.progsoc.uts.edu.au 138.25.6.2
New Machine (Before Switch): phobos.progsoc.uts.edu.au 138.25.6.14

New Machine (After Switch): orgo.progsoc.uts.edu.au 138.25.6.2
Old Machine (After Switch): phobos.progsoc.uts.edu.au 138.25.6.14

Now, on any host within our network (but not external to the network) we've got major DNS issues.
From one of our machines (incubus)

jimmyk@incubus:~$ nslookup orgo
Server:         138.25.6.2
Address:        138.25.6.2#53

Name:   orgo.progsoc.uts.edu.au
Address: 138.25.6.2

jimmyk@incubus:~$ nslookup phobos
Server:         138.25.6.2
Address:        138.25.6.2#53

Name:   phobos.progsoc.uts.edu.au
Address: 138.25.6.14

But now when I ssh to orgo (from incubus, or any other machine on the network including orgo and phobos), it sends me to phobos.
When I ssh to phobos, it goes to the correct machine (ie. Phobos).

Anyone got suggestions on how this might be caused?

Did you modify the IPs in the /etc/hosts on the two servers?

Did verify that the two IPs are NOT in /etc/hosts on your other servers?

Originally Posted by jlightner Did you modify the IPs in the /etc/hosts on the two servers?

Yea, /etc/hosts for phobos (along with the usual IPV6 stuff):

138.25.6.14     phobos.progsoc.uts.edu.au phobos

for orgo:

138.25.6.2      orgo.progsoc.uts.edu.au orgo

Originally Posted by jlightner Did verify that the two IPs are NOT in /etc/hosts on your other servers?

Yes, /etc/hosts contains no reference to any other machines on all servers.

The resolv.conf on your non-dns servers point to the IPs or the names? If the names that would be your problem since you changed the name of the DNS server.

Did you stop named on the old server?

Did you modify the zone file(s) on the new server to include the correct IP for it (including the serial number) then bounce named there afterwards?

Originally Posted by jlightner The resolv.conf on your non-dns servers point to the IPs or the names? If the names that would be your problem since you changed the name of the DNS server.

They all point to IP's

Did you stop named on the old server?

Yes

Did you modify the zone file(s) on the new server to include the correct IP for it (including the serial number) then bounce named there afterwards?

What do you mean by bounce?
Which zone file(s) should I modify if any? They're all point to orgo at 138.25.6.2 which is the correct IP address.

Rereading your original post though you say IPs it looks like you changed the host names as well. My comment was based on the thought you changed only IPs but I see now that may not have been the case.

The zone files to be changes would be the ones that contain reference to either the IPs or the hostnames for orgo and phobos. (You may have reverse lookup zone files as well.)

The serial number in a zone file should be changed whenever you change any other information within the file so that other servers querying it know to use the updated information it provides rather than any information they may have cached previously.

Bouncing named (assuming you're doing BIND) makes named itself reread any information you have in named.conf or the zone files. By "bounce" I mean stop named then start named.

Also do you have any slave/cache DNS servers? If so you may need to clean the cache on those and bounce named there to reload the information from the master DNS server.

Originally Posted by jlightner The zone files to be changes would be the ones that contain reference to either the IPs or the hostnames for orgo and phobos. (You may have reverse lookup zone files as well.) The serial number in a zone file should be changed whenever you change any other information within the file so that other servers querying it know to use the updated information it provides rather than any information they may have cached previously.

I've updated the serial number just for the sake of seeing if it makes a difference, however it appears to not have made a change at all.

Bouncing named (assuming you're doing BIND) makes named itself reread any information you have in named.conf or the zone files. By "bounce" I mean stop named then start named.

I've restarted named on a few occasions, so that doesn't seem to be making a difference.

Also do you have any slave/cache DNS servers? If so you may need to clean the cache on those and bounce named there to reload the information from the master DNS server.

Not that I know of for the machines on the local network. (I've only been in this sysadmin position for about 3 weeks - and I'm quite out of touch with the workings of the network).

Are you sure this is a dns issue at all?
What happens if you try to ssh to the IP?
Do you have the problem only with ssh, or anything (like ping,...)?

Originally Posted by rupertwh Are you sure this is a dns issue at all?

It may not be a DNS issue - but I thought that would be a good place to start. Some people on our societies admin mailing list said that 'DNS is @@@@@@ (fill in the blank), and so that's where I thought I'd start.
Personally, I don't think its DNS - but I'm not sure what else could be causing it.

What happens if you try to ssh to the IP? Do you have the problem only with ssh, or anything (like ping,...)?

If I SSH to the IP it goes to the correct machine. Ping seems to be pining the right IP too.

The fact that this problem is network wide (and not just related to one machine) however, makes me think it could be something to do with DNS.

Are the users in question on Windoze boxen?

Windoze boxen cache DNS.

Try
Start --> Run
Type in "command" and hit enter
In window that opens type "ipconfig /flushdns"

Test to see if that Windoze box still has the issue.

All the machines in question are either running Debian (in the case of Incubus and a few other machines), Ubuntu, or MacOS X Tiger (with all patches applied).

I've tried flushing the DNS on a few machines with no success in solving the problem.

One thing for anybody to note: The dns info has *not* changed! The names still resolve to the same IPs as before. Only the underlying hardware was swapped. So from a network point of view, the MACs are all that have changed.

Swapping hardware can create confusion through the arp cache. (Though actually not that long and usually only in terms of hosts not responding.)
But just to be on the safe side, can you please examine the output of 'arp' on the clients and confirm that the information is correct?

Are you *really* sure that ssh sends you to the wrong machine? How exactly did you determine that? Are you sure you weren't confused by e.g. some leftover old names on the respective servers?

Then again, ssh should have complained that current keys do not match the ones in known_hosts.

Can you please confirm that there is difference between pinging by name and pinging by ip? (Use tcpdump on servers to see where pings are actually going)

Regards

Rupert

Originally Posted by rupertwh Swapping hardware can create confusion through the arp cache. (Though actually not that long and usually only in terms of hosts not responding.) But just to be on the safe side, can you please examine the output of 'arp' on the clients and confirm that the information is correct?

arp on incubus gives out the following:

Address                  HWtype  HWaddress           Flags Mask            Iface
mephisto.progsoc.uts.ed  ether   00:40:8C:61:5A:60   C                     eth0
medusa-ps6.progsoc.uts.  ether   00:50:BA:61:87:40   C                     eth0
orgo.progsoc.uts.edu.au  ether   00:E0:4C:81:A3:35   C                     eth0

The mac address given for orgo matches that on the machine that is the 'real' orgo.

Are you *really* sure that ssh sends you to the wrong machine? How exactly did you determine that? Are you sure you weren't confused by e.g. some leftover old names on the respective servers?

Yea, I'm *really* sure. On our old machine (ie the one we're getting sent to), we have a message coming up saying 'this is the old Orgo, your previous home directories are located at............' etc. We have no such message being produced on login on the new Orgo.

Then again, ssh should have complained that current keys do not match the ones in known_hosts.

This has occoured.

Can you please confirm that there is difference between pinging by name and pinging by ip? (Use tcpdump on servers to see where pings are actually going)

tcpdump is spitting out far too much information for me to get anything meaningful out of it - any suggestions on how to filter the results?

Originally Posted by Jamesk5 tcpdump is spitting out far too much information for me to get anything meaningful out of it - any suggestions on how to filter the results?

Try something like

tcpdump -l -i eth0 icmp

or

tcpdump -l -i eth0 icmp and host <hostname or ip address>

From your info so far, the following can be concluded:

a) It CANNOT really be a dns issue, because dns info wasn't changed.
b) It CAN ONLY be a dns issue, because ssh works via ip, but not via host name.

Which is weird all right... ;o)

In the process of swapping the two machines, was dns info maybe changed temporarily?

Rupert

Originally Posted by rupertwh Try something like tcpdump -l -i eth0 icmp or tcpdump -l -i eth0 icmp and host <hostname or ip address>

When I ping either the IP address (138.25.6.2) or the hostname orgo (or even orgo.progsoc.uts.edu.au), it goes to the correct machine.

In the process of swapping the two machines, was dns info maybe changed temporarily?

What info are you referring to? DNS records, or something different?