LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices



Reply
 
Search this Thread
Old 06-04-2007, 08:42 AM   #1
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Rep: Reputation: 15
Weird DNS Issue


I recently migrated our DNS/LDAP/NFS server to a new machine. The Old machine has taken the IP address of what now is the New machine. IE:
Old Machine (Before Switch): orgo.progsoc.uts.edu.au 138.25.6.2
New Machine (Before Switch): phobos.progsoc.uts.edu.au 138.25.6.14

New Machine (After Switch): orgo.progsoc.uts.edu.au 138.25.6.2
Old Machine (After Switch): phobos.progsoc.uts.edu.au 138.25.6.14

Now, on any host within our network (but not external to the network) we've got major DNS issues.
From one of our machines (incubus)
Code:
jimmyk@incubus:~$ nslookup orgo
Server:         138.25.6.2
Address:        138.25.6.2#53

Name:   orgo.progsoc.uts.edu.au
Address: 138.25.6.2
Code:
jimmyk@incubus:~$ nslookup phobos
Server:         138.25.6.2
Address:        138.25.6.2#53

Name:   phobos.progsoc.uts.edu.au
Address: 138.25.6.14
But now when I ssh to orgo (from incubus, or any other machine on the network including orgo and phobos), it sends me to phobos.
When I ssh to phobos, it goes to the correct machine (ie. Phobos).

Anyone got suggestions on how this might be caused?
 
Old 06-04-2007, 08:53 AM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Did you modify the IPs in the /etc/hosts on the two servers?

Did verify that the two IPs are NOT in /etc/hosts on your other servers?
 
Old 06-04-2007, 09:06 AM   #3
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by jlightner
Did you modify the IPs in the /etc/hosts on the two servers?
Yea, /etc/hosts for phobos (along with the usual IPV6 stuff):
Code:
138.25.6.14     phobos.progsoc.uts.edu.au phobos
for orgo:
Code:
138.25.6.2      orgo.progsoc.uts.edu.au orgo
Quote:
Originally Posted by jlightner
Did verify that the two IPs are NOT in /etc/hosts on your other servers?
Yes, /etc/hosts contains no reference to any other machines on all servers.
 
Old 06-04-2007, 09:53 AM   #4
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
The resolv.conf on your non-dns servers point to the IPs or the names? If the names that would be your problem since you changed the name of the DNS server.

Did you stop named on the old server?

Did you modify the zone file(s) on the new server to include the correct IP for it (including the serial number) then bounce named there afterwards?
 
Old 06-04-2007, 10:03 AM   #5
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by jlightner
The resolv.conf on your non-dns servers point to the IPs or the names? If the names that would be your problem since you changed the name of the DNS server.
They all point to IP's

Quote:
Did you stop named on the old server?
Yes

Quote:
Did you modify the zone file(s) on the new server to include the correct IP for it (including the serial number) then bounce named there afterwards?
What do you mean by bounce?
Which zone file(s) should I modify if any? They're all point to orgo at 138.25.6.2 which is the correct IP address.
 
Old 06-04-2007, 10:42 AM   #6
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Rereading your original post though you say IPs it looks like you changed the host names as well. My comment was based on the thought you changed only IPs but I see now that may not have been the case.

The zone files to be changes would be the ones that contain reference to either the IPs or the hostnames for orgo and phobos. (You may have reverse lookup zone files as well.)

The serial number in a zone file should be changed whenever you change any other information within the file so that other servers querying it know to use the updated information it provides rather than any information they may have cached previously.

Bouncing named (assuming you're doing BIND) makes named itself reread any information you have in named.conf or the zone files. By "bounce" I mean stop named then start named.

Also do you have any slave/cache DNS servers? If so you may need to clean the cache on those and bounce named there to reload the information from the master DNS server.
 
Old 06-05-2007, 01:32 AM   #7
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by jlightner
The zone files to be changes would be the ones that contain reference to either the IPs or the hostnames for orgo and phobos. (You may have reverse lookup zone files as well.)

The serial number in a zone file should be changed whenever you change any other information within the file so that other servers querying it know to use the updated information it provides rather than any information they may have cached previously.
I've updated the serial number just for the sake of seeing if it makes a difference, however it appears to not have made a change at all.

Quote:
Bouncing named (assuming you're doing BIND) makes named itself reread any information you have in named.conf or the zone files. By "bounce" I mean stop named then start named.
I've restarted named on a few occasions, so that doesn't seem to be making a difference.

Quote:
Also do you have any slave/cache DNS servers? If so you may need to clean the cache on those and bounce named there to reload the information from the master DNS server.
Not that I know of for the machines on the local network. (I've only been in this sysadmin position for about 3 weeks - and I'm quite out of touch with the workings of the network).
 
Old 06-05-2007, 01:11 PM   #8
rupertwh
Member
 
Registered: Sep 2006
Location: Munich, Germany
Distribution: Debian / Ubuntu
Posts: 297

Rep: Reputation: 49
Are you sure this is a dns issue at all?
What happens if you try to ssh to the IP?
Do you have the problem only with ssh, or anything (like ping,...)?
 
Old 06-06-2007, 12:18 AM   #9
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by rupertwh
Are you sure this is a dns issue at all?
It may not be a DNS issue - but I thought that would be a good place to start. Some people on our societies admin mailing list said that 'DNS is @@@@@@ (fill in the blank), and so that's where I thought I'd start.
Personally, I don't think its DNS - but I'm not sure what else could be causing it.

Quote:
What happens if you try to ssh to the IP?
Do you have the problem only with ssh, or anything (like ping,...)?
If I SSH to the IP it goes to the correct machine. Ping seems to be pining the right IP too.

The fact that this problem is network wide (and not just related to one machine) however, makes me think it could be something to do with DNS.

Last edited by Jamesk5; 06-06-2007 at 12:27 AM.
 
Old 06-06-2007, 08:47 AM   #10
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Are the users in question on Windoze boxen?

Windoze boxen cache DNS.

Try
Start --> Run
Type in "command" and hit enter
In window that opens type "ipconfig /flushdns"

Test to see if that Windoze box still has the issue.
 
Old 06-06-2007, 09:02 AM   #11
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
All the machines in question are either running Debian (in the case of Incubus and a few other machines), Ubuntu, or MacOS X Tiger (with all patches applied).

I've tried flushing the DNS on a few machines with no success in solving the problem.
 
Old 06-06-2007, 09:31 AM   #12
rupertwh
Member
 
Registered: Sep 2006
Location: Munich, Germany
Distribution: Debian / Ubuntu
Posts: 297

Rep: Reputation: 49
One thing for anybody to note: The dns info has *not* changed! The names still resolve to the same IPs as before. Only the underlying hardware was swapped. So from a network point of view, the MACs are all that have changed.

Swapping hardware can create confusion through the arp cache. (Though actually not that long and usually only in terms of hosts not responding.)
But just to be on the safe side, can you please examine the output of 'arp' on the clients and confirm that the information is correct?

Are you *really* sure that ssh sends you to the wrong machine? How exactly did you determine that? Are you sure you weren't confused by e.g. some leftover old names on the respective servers?

Then again, ssh should have complained that current keys do not match the ones in known_hosts.

Can you please confirm that there is difference between pinging by name and pinging by ip? (Use tcpdump on servers to see where pings are actually going)

Regards

Rupert
 
Old 06-06-2007, 10:05 AM   #13
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by rupertwh
Swapping hardware can create confusion through the arp cache. (Though actually not that long and usually only in terms of hosts not responding.)
But just to be on the safe side, can you please examine the output of 'arp' on the clients and confirm that the information is correct?
arp on incubus gives out the following:
Code:
Address                  HWtype  HWaddress           Flags Mask            Iface
mephisto.progsoc.uts.ed  ether   00:40:8C:61:5A:60   C                     eth0
medusa-ps6.progsoc.uts.  ether   00:50:BA:61:87:40   C                     eth0
orgo.progsoc.uts.edu.au  ether   00:E0:4C:81:A3:35   C                     eth0
The mac address given for orgo matches that on the machine that is the 'real' orgo.

Quote:
Are you *really* sure that ssh sends you to the wrong machine? How exactly did you determine that? Are you sure you weren't confused by e.g. some leftover old names on the respective servers?
Yea, I'm *really* sure. On our old machine (ie the one we're getting sent to), we have a message coming up saying 'this is the old Orgo, your previous home directories are located at............' etc. We have no such message being produced on login on the new Orgo.

Quote:
Then again, ssh should have complained that current keys do not match the ones in known_hosts.
This has occoured.

Quote:
Can you please confirm that there is difference between pinging by name and pinging by ip? (Use tcpdump on servers to see where pings are actually going)
tcpdump is spitting out far too much information for me to get anything meaningful out of it - any suggestions on how to filter the results?
 
Old 06-06-2007, 07:00 PM   #14
rupertwh
Member
 
Registered: Sep 2006
Location: Munich, Germany
Distribution: Debian / Ubuntu
Posts: 297

Rep: Reputation: 49
Quote:
Originally Posted by Jamesk5
tcpdump is spitting out far too much information for me to get anything meaningful out of it - any suggestions on how to filter the results?
Try something like
Code:
tcpdump -l -i eth0 icmp
or
Code:
tcpdump -l -i eth0 icmp and host <hostname or ip address>
From your info so far, the following can be concluded:

a) It CANNOT really be a dns issue, because dns info wasn't changed.
b) It CAN ONLY be a dns issue, because ssh works via ip, but not via host name.

Which is weird all right... ;o)

In the process of swapping the two machines, was dns info maybe changed temporarily?

Rupert
 
Old 06-06-2007, 08:26 PM   #15
Jamesk5
LQ Newbie
 
Registered: Feb 2005
Location: Australia
Distribution: Debian, CentOS
Posts: 25

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by rupertwh
Try something like
Code:
tcpdump -l -i eth0 icmp
or
Code:
tcpdump -l -i eth0 icmp and host <hostname or ip address>
When I ping either the IP address (138.25.6.2) or the hostname orgo (or even orgo.progsoc.uts.edu.au), it goes to the correct machine.

Quote:
In the process of swapping the two machines, was dns info maybe changed temporarily?
What info are you referring to? DNS records, or something different?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Weird DNS problem rikos.k Linux - Server 3 04-02-2007 10:00 AM
a weird issue about which jiawj Red Hat 2 04-26-2005 10:06 AM
Weird DNS (?) behaviour jalonsom Linux - Networking 1 10-31-2004 07:55 AM
weird DNS related (?) issue bobcat Linux - Networking 1 06-10-2002 09:08 PM
really weird DNS issue. Help needed ASAP! Supp0rtLinux Linux - Networking 2 05-20-2002 11:02 PM


All times are GMT -5. The time now is 06:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration