LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices



Reply
 
Search this Thread
Old 06-15-2011, 02:52 PM   #1
iambrucelee
LQ Newbie
 
Registered: May 2009
Posts: 4

Rep: Reputation: 4
slow dns on rhel6 with ipv6 going through a firewall (yum ssh firefox)


I wanted to post this here to help anyone that might have noticed any sort of performance issues with a RHEL 6 box. Im sure this issue will become a lot more prevelant when CentOS 6 comes out. It took me days of troubleshooting to figure this out and hopefully this will save a headache for others. These issues are also present on Fedora 10 and Fedora 11. I've seen quite a few forum posts on it already. (ie http://www.linuxquestions.org/questi...a-11-a-778069/)

Skip to the bottom for the solution.

Here are the symptoms:

-ssh to the machine takes a long time before you finally get in. This usually points to DNS issues.
-dig and host succeed and resolve names very fast. (few ms)
-telnet to a port takes a longer than usual.
-Firefox is slow
-Yum is slow

Just on a hunch I disabled ipv6 and performance improved, but was still a lot slower than usual for certain applications.

After a bunch of troubleshooting here's what I discovered:

1. RHEL5 works perfectly. (ssh, telnet are fast)
2. this only happens when you make a DNS query through a firewall. If you have a DNS server on the same network segment, its super fast.
3. changing the timeout in /etc/resolv.conf helps, but its still not as fast as RHEL5 or Debian machines.

tcpdumps showed me that even with ipv6 disabled, AAAA queries were still happening for ssh, yum and whois. So something must have changed from RHEL5 to RHEL6.

Redhat's knowledge base was of no help what so ever.

After reading through hundreds of comments in multiple bug reports, I finally discovered the root cause:

Somewhere down the line, the maintainer for glibc decided to change the behavior of how the DNS resolver works. Now instead of opening a socket for each request, it uses the same socket for a A and AAAA request. A lot of hardware (firewalls, etc) gets confused and only sends back 1 reply. In return, your machine sits there and waits for the 2nd reply. (https://bugzilla.redhat.com/show_bug.cgi?id=505105)

The glibc maintainers solution was to "fix the broken hardware". In an enterprise environment, I dont really have an option to upgrade my firewall today, even if theres a patch for it. I'm running a Juniper SSG-550 firewall, so Im sure many other people are having the same issues. And I have a bunch of enterprise RHEL 6 servers I need to deploy. The work around for it was to install a local dns caching server on the machine itself which is ludicrous. They're telling me that I have to install a local caching server on each and every RHEL 6 server I have? WTF?
(https://bugzilla.redhat.com/show_bug.cgi?id=459756)

Finally he decided to implement a fallback option thats not really documented anywhere except here: http://sourceware.org/ml/libc-alpha/.../msg00063.html

If two requests from the same port are not handled correctly close the socket and open a new one before sending the second request. The 'single-request-reopen' option in /etc/resolv.conf can be used to select this mode right away, instead of rediscovering the necessity is every process again.

So in short heres what I did and it resolved my issue:

place this In your /etc/resolv.conf

options single-request-reopen
 
Old 06-15-2011, 04:23 PM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Thanks for posting this. I hadn't run into the issue yet but suspect I would eventually.

Please go to Thread Tools at top and mark this solved. It helps others when doing web searches to find resolved issues more quickly since it will show up in the thread title.
 
Old 06-29-2011, 01:48 PM   #3
rryder
LQ Newbie
 
Registered: Jun 2011
Posts: 2

Rep: Reputation: Disabled
Thumbs up

Hi! Thanks for pointing out this wasn't in the Red Hat kbase! I think customers will benefit from this, so please check out https://access.redhat.com/kb/docs/DOC-58626
 
Old 06-29-2011, 04:16 PM   #4
erinn
LQ Newbie
 
Registered: Jun 2011
Posts: 3

Rep: Reputation: Disabled
I put a bug in a kernel.org bugzilla report requesting documentation for the option in resolv.conf here: https://bugzilla.kernel.org/show_bug.cgi?id=38542

As well, I opened a request with Red Hat to include said documentation here: https://bugzilla.redhat.com/show_bug.cgi?id=717770

Thanks a lot for posting this up, I had been struggling with this issue as well.

-Erinn
 
Old 06-29-2011, 05:46 PM   #5
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,919

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
Quote:
Originally Posted by iambrucelee View Post

Here are the symptoms:

-ssh to the machine takes a long time before you finally get in. This usually points to DNS issues.
-dig and host succeed and resolve names very fast. (few ms)
-telnet to a port takes a longer than usual.
-Firefox is slow
-Yum is slow
Thank you very much for posting that. I've noticed a certain unfortunate interaction between some browsers and squid, which seems to have some elements in common with what you have - not the same, but similar effects - so your work gives me somewhere to start. I hadn't even thought of glibc, to be honest.

Quote:
The work around for it was to install a local dns caching server on the machine itself which is ludicrous. They're telling me that I have to install a local caching server on each and every RHEL 6 server I have?
Presumably, they are saying that you need one DNS cache on the 'local' side of the firewall, rather than one on each machine? Still a work-around for a broken situation, though.
 
Old 02-24-2012, 09:31 AM   #6
crnkyadm
LQ Newbie
 
Registered: Feb 2012
Posts: 1

Rep: Reputation: Disabled
FYI: not just through firewalls

Hey, I just wanted to let you know that this happens even when there is no firewall between client and server. I'm building a Oracle Linux 6.2 (RHEL knock-off like CentOS) virtual machine running on ESXi 4.1 with Cisco Nexus network infrastructure and had this issue. About the only thing that I can think of is that our DNS servers are behind Cisco load-balancers, so the LB may be tripping up on the traffic like the firewalls seem to. I changed the resolv.conf to point directly to the DNS servers and the result seems to support the idea that the LB is getting tripped up.
 
Old 09-10-2013, 11:38 AM   #7
crackptb
LQ Newbie
 
Registered: Sep 2013
Posts: 2

Rep: Reputation: Disabled
I for a change, have found issue sligthly elswhere... Not on the server I am trying to access but Linux box I am using everyday.
I have found that delay was caused by GSSAPI authentication method on SSH local client. To resolve the issue I have edited /etc/ssh/ssh_config and updated line -> GSSAPIAuthentication no

This mod solved the speed issue for me as I use only ssh key exchange or manually typed passwords.

Last edited by crackptb; 09-10-2013 at 11:45 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
DNAT and SNAT for ipv6 in RHEL6 marifran Linux - Kernel 0 12-09-2010 05:09 AM
Firefox still slow even with IPv6 disabled... Biggen Slackware 6 07-22-2007 04:58 PM
Slow SSH, how do I set up a proper DNS config? billy3 Linux - General 3 05-03-2007 04:57 PM
Wireless Slow.. possible DNS .. ipv6 disabled EclipseAgent Suse/Novell 0 08-05-2006 11:33 PM
Slow DNS search +firefox +FC5 Kosmaty Linux - Networking 8 06-02-2006 11:44 AM


All times are GMT -5. The time now is 01:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration