LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 08-16-2008, 07:37 PM   #1
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Rep: Reputation: 30
nss_ldap hangs at failover of master LDAP server to slave


Hello all,

I have two LDAP servers running openLDAP. Both have replication working fine. However, when the master fails, the switchover to the slave is incredibly slow. Performing an strace on finger reveals that at every command, an attempt to query the master is performed. The amount of time it waits is dictated by bind_timelimit in /etc/ldap.conf. That is, nothing records that the master is down; every finger still tries to query the master. With a default bind_timelimit of 120 seconds, this is horrible. Even with 1 second this is bad, since that is relly just a workaround.

Another curiosity is that bind_timelimit seems to have no effect if the LDAP servers are listed in /etc/openldap/ldap.conf. Only those in /etc/ldap.conf honor bind_timelimit.

I should note that I am running nscd. However, it doesn't seem to be doing a whole lot in this case...

Is there some way to essentially tell everything that the master is down?
 
Old 08-17-2008, 12:22 AM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,965
Blog Entries: 11

Rep: Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865
You'll probably want to start looking at linux-HA in
this situation ... http://linux-ha.org/


Cheers,
Tink
 
Old 08-17-2008, 09:00 AM   #3
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Original Poster
Rep: Reputation: 30
I agree that that's the best solution. However, it's essentially impossible in our network setup. The network is designed in such a way that virtual IPs are disallowed - every IP must have a unique MAC address assigned to it. Otherwise, things get blocked. Everything I've read on about HA anything involves virtual IPs, whether heartbeat or CARP is being used to determine who is alive.

The only workaround I can think of to that involves connecting secondary NICs but purposely leaving them unconfigured. At master failure, the slave configures the second NIC instead of creating a virtual IP. This way, it has a unique IP and MAC address (there are no virtual MAC addresses). Granted, this should work, but I find it a tad ugly/hackish.

I feel like there must exist a single daemon that can do all the host checking, instead of every single command that uses LDAP. LDAP is supposed to have redundancy built in, after all.
 
Old 08-17-2008, 09:04 AM   #4
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Original Poster
Rep: Reputation: 30
Staring at my last post...

I thought of a way to do exactly that with a very short perl script. It would contain a list of the LDAP servers, and would ping them. It would dynamically edit /etc/ldap.conf based upon this information.

If I don't get any responses / there isn't a way to do what I'm asking, then I'll write and post the script for all to enjoy.
 
Old 08-17-2008, 01:11 PM   #5
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,965
Blog Entries: 11

Rep: Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865
The other alternative would be to have a script
check the ldap-servers from the DNS box and
dynamically change the order of the DNS entries.
 
Old 08-17-2008, 04:06 PM   #6
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Original Poster
Rep: Reputation: 30
Finished writing the script. It does work in my setup, but there is no warranty, of course.

If others are to use this, then LDAP_PORT, LDAP_PREFIX, LDAP_HOSTS, LDAP_CONF_PATH, and TIMEOUT should be adjusted as needed.

The DNS idea is a much cooler solution, since that would only have to run on one system versus all of those that use LDAP. Unfortunately, I don't have access to our DNS servers (kinda like I can't use virtual IPs...)

Thanks for the help!

Code:
#!/usr/bin/perl -w

# This script updates /etc/ldap.conf based upon which
# LDAP servers are up or not.  This is so utilities like
# finger do not query the LDAP servers which are down

use strict;
use IO::Socket;

# define some constants ---------------------------------------
my $LDAP_PORT = 636; # port used by LDAP; I use secure LDAP
my $LDAP_PREFIX = "ldaps://"; # prefix for an LDAP URI
	                      # use ldap:// for normal LDAP
my $TIMEOUT = 5; # length of time, in seconds, until we 
		 # give up on a host
my @LDAP_HOSTS = ( "fake1.host.com",
	           "fake2.host.com" );
	           # The hosts which have LDAP running.
	           # Note that the first host listed is assumed
	           # to be the master
my $LDAP_CONF_PATH = "/etc/ldap.conf";
# end constant definitions-------------------------------------


# begin subroutine definitions---------------------------------

# tests to see if a host is alive
# takes the host name as a param
# returns 1 if alive, else 0
sub is_host_alive($) 
{
	my $host = shift();
	my $retval = 0;
	my $sock; # socket to test life

	$sock = new IO::Socket::INET(
		PeerAddr => $host,
		PeerPort => $LDAP_PORT,
		Proto => 'tcp',
		Timeout => $TIMEOUT
		);

	if ( $sock ) {
		$retval = 1;
		close( $sock );
	}

	return $retval;
}

# changes the URI line in the LDAP config file
# takes the whole URI line as a param
sub change_uri($) 
{
	my $uri = shift();
	my $CONF; # filehandle for config file
	my @lines; # the lines in the config file
	my $change = 0; # set to 1 if the uri line differs

	# try to open the file
	open( CONF, "<$LDAP_CONF_PATH" ) or die
		"Failed to open $LDAP_CONF_PATH for reading: $!";

	# read in the lines
	@lines = <CONF>;
	close( CONF );

	# find and change the uri line
	for( my $x = 0; $x <= $#lines; $x++ ) {
		if ( $lines[ $x ] =~ /^uri/ &&
		     $lines[ $x ] ne "$uri\n" ) {
			$change = 1;
			$lines[ $x ] = "$uri\n";
		}
	}

	# write out the config file, if needed
	if ( $change ) {
		open( CONF, ">$LDAP_CONF_PATH" ) or die
			"Failed to open $LDAP_CONF_PATH for writing: $!";
		print CONF @lines;
		close( CONF );
	}
}

# end subroutine definitions-----------------------------------


# main program code--------------------------------------------

my @alive_hosts; # hosts which respond
my $uri_line; # the new uri line

foreach my $current ( @LDAP_HOSTS ) {
	if ( is_host_alive( $current ) ) {
		push( @alive_hosts, $current );
	}
}

# make the uri line
$uri_line = "uri ";
foreach my $current ( @alive_hosts ) {
	$uri_line .= "$LDAP_PREFIX$current ";
}

# change the config file
change_uri( $uri_line );

# note that this must be run from cron or equivalent;
# it only runs once
 
Old 08-17-2008, 05:00 PM   #7
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,965
Blog Entries: 11

Rep: Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865
Heh ... well, yet another alternative would then have been (assuming that your
clients use /etc/hosts and /etc/host.conf has the order files,bind) your script
could be changing the hosts file instead ...
 
Old 06-21-2009, 10:20 AM   #8
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Original Poster
Rep: Reputation: 30
In case anyone is actually using / will use the posted script...

This morning, something ugly happened on our master LDAP server. It never actually went down; it could be pinged, and ports for provided services were still opened. However, those services, including LDAP, would never actually do anything. The above script only checks for the port, so it didn't catch that (much to the dismay of some users...)

I have posted below a revised version that actually checks to see if the server can respond or not. Again, no warranties, redistribute if you want, etc. It requires that Net::LDAPS is installed.

Code:
#!/usr/bin/perl -w

# This script updates /etc/ldap.conf based upon which
# LDAP servers are up or not.  This is so utilities like
# finger do not query the LDAP servers which are down

# Note that the is_host_alive routine is based upon
# De Bodt Lieven's code from 
# http://www.monitoringexchange.org/cgi-bin/page.cgi?g=Detailed%2F1325.html;d=1

use strict;
use Net::LDAPS;

# define some constants ---------------------------------------
my $LDAP_PORT = 636; # port used by LDAP; I use secure LDAP
my $LDAP_VERSION = 3; # version of LDAP to test for
my $LDAP_PREFIX = "ldaps://"; # prefix for an LDAP URI
	                      # use ldap:// for normal LDAP
my $TIMEOUT = 5; # length of time, in seconds, until we 
		 # give up on a host
my @LDAP_HOSTS = ( "fake1.host.com",
	           "fake2.host.com" );
	           # The hosts which have LDAP running.
	           # Note that the first host listed is assumed
	           # to be the master
my $LDAP_CONF_PATH = "/etc/ldap.conf";
# end constant definitions-------------------------------------


# begin subroutine definitions---------------------------------

# tests to see if a host is alive
# takes the host name as a param
# returns 1 if alive, else 0
sub is_host_alive($) 
{
	my $host = shift();
	my $retval = 0;
	my $ldap; # session with the LDAP server

	# create the session
	$ldap = Net::LDAPS->new( $host, 
				 port => $LDAP_PORT,
				 version => 3,
				 timeout => $TIMEOUT );
	if( $ldap ) {
		my $resp; # response from LDAP server
		$resp = $ldap->bind();
		unless( $resp->code ) {
			# connection successful
			$resp = $ldap->unbind(); #disconnect
			$retval = 1;
		}
	}

	return $retval;
}

# changes the URI line in the LDAP config file
# takes the whole URI line as a param
sub change_uri($) 
{
	my $uri = shift();
	my $CONF; # filehandle for config file
	my @lines; # the lines in the config file
	my $change = 0; # set to 1 if the uri line differs

	# try to open the file
	open( CONF, "<$LDAP_CONF_PATH" ) or die
		"Failed to open $LDAP_CONF_PATH for reading: $!";

	# read in the lines
	@lines = <CONF>;
	close( CONF );

	# find and change the uri line
	for( my $x = 0; $x <= $#lines; $x++ ) {
		if ( $lines[ $x ] =~ /^uri/ &&
		     $lines[ $x ] ne "$uri\n" ) {
			$change = 1;
			$lines[ $x ] = "$uri\n";
		}
	}

	# write out the config file, if needed
	if ( $change ) {
		open( CONF, ">$LDAP_CONF_PATH" ) or die
			"Failed to open $LDAP_CONF_PATH for writing: $!";
		print CONF @lines;
		close( CONF );
	}
}

# end subroutine definitions-----------------------------------


# main program code--------------------------------------------

my @alive_hosts; # hosts which respond
my $uri_line; # the new uri line

foreach my $current ( @LDAP_HOSTS ) {
	if ( is_host_alive( $current ) ) {
		push( @alive_hosts, $current );
	}
}

# make the uri line
$uri_line = "uri ";
foreach my $current ( @alive_hosts ) {
	$uri_line .= "$LDAP_PREFIX$current ";
}

# change the config file
change_uri( $uri_line );

# note that this must be run from cron or equivalent;
# it only runs once
 
Old 03-31-2010, 10:27 AM   #9
mrlondon
LQ Newbie
 
Registered: Aug 2008
Posts: 2

Rep: Reputation: 0
Did you find out why your ldap server was hanging?

Our main ldap server has hung twice in the last few weeks (a Windows server). Nothing in the logs shows that anything was wrong with it. It was pingable, but couldn't be connected to. Did you ever find the reason why your ldap server was hanging? Thanks. - Mark
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LDAP in master-slave replication aravind1024004 Linux - Networking 1 04-20-2008 12:18 PM
LDAP in master-slave replication aravind1024004 Linux - Server 1 04-10-2008 10:21 PM
Ldap Master slave aravindhcl Linux - Server 1 12-26-2007 07:26 PM
ldap replication master - slave jadid Linux - Software 2 10-18-2006 11:25 AM
nss_ldap, can't contact LDAP server! mesh2005 Linux - Networking 3 12-06-2005 01:22 AM


All times are GMT -5. The time now is 01:37 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration