nss_ldap hangs at failover of master LDAP server to slave

linux=future · 08-16-2008, 07:37 PM

Hello all,

I have two LDAP servers running openLDAP. Both have replication working fine. However, when the master fails, the switchover to the slave is incredibly slow. Performing an strace on finger reveals that at every command, an attempt to query the master is performed. The amount of time it waits is dictated by bind_timelimit in /etc/ldap.conf. That is, nothing records that the master is down; every finger still tries to query the master. With a default bind_timelimit of 120 seconds, this is horrible. Even with 1 second this is bad, since that is relly just a workaround.

Another curiosity is that bind_timelimit seems to have no effect if the LDAP servers are listed in /etc/openldap/ldap.conf. Only those in /etc/ldap.conf honor bind_timelimit.

I should note that I am running nscd. However, it doesn't seem to be doing a whole lot in this case...

Is there some way to essentially tell everything that the master is down?

Tinkster · 08-17-2008, 12:22 AM

You'll probably want to start looking at linux-HA in
this situation ... http://linux-ha.org/

Cheers,
Tink

linux=future · 08-17-2008, 09:00 AM

I agree that that's the best solution. However, it's essentially impossible in our network setup. The network is designed in such a way that virtual IPs are disallowed - every IP must have a unique MAC address assigned to it. Otherwise, things get blocked. Everything I've read on about HA anything involves virtual IPs, whether heartbeat or CARP is being used to determine who is alive.

The only workaround I can think of to that involves connecting secondary NICs but purposely leaving them unconfigured. At master failure, the slave configures the second NIC instead of creating a virtual IP. This way, it has a unique IP and MAC address (there are no virtual MAC addresses). Granted, this should work, but I find it a tad ugly/hackish.

I feel like there must exist a single daemon that can do all the host checking, instead of every single command that uses LDAP. LDAP is supposed to have redundancy built in, after all.

linux=future · 08-17-2008, 09:04 AM

Staring at my last post...

I thought of a way to do exactly that with a very short perl script. It would contain a list of the LDAP servers, and would ping them. It would dynamically edit /etc/ldap.conf based upon this information.

If I don't get any responses / there isn't a way to do what I'm asking, then I'll write and post the script for all to enjoy.

Tinkster · 08-17-2008, 01:11 PM

The other alternative would be to have a script
check the ldap-servers from the DNS box and
dynamically change the order of the DNS entries.

linux=future · 08-17-2008, 04:06 PM

Finished writing the script. It does work in my setup, but there is no warranty, of course.

If others are to use this, then LDAP_PORT, LDAP_PREFIX, LDAP_HOSTS, LDAP_CONF_PATH, and TIMEOUT should be adjusted as needed.

The DNS idea is a much cooler solution, since that would only have to run on one system versus all of those that use LDAP. Unfortunately, I don't have access to our DNS servers (kinda like I can't use virtual IPs...)

Thanks for the help!

Code:

#!/usr/bin/perl -w

# This script updates /etc/ldap.conf based upon which
# LDAP servers are up or not.  This is so utilities like
# finger do not query the LDAP servers which are down

use strict;
use IO::Socket;

# define some constants ---------------------------------------
my $LDAP_PORT = 636; # port used by LDAP; I use secure LDAP
my $LDAP_PREFIX = "ldaps://"; # prefix for an LDAP URI
	                      # use ldap:// for normal LDAP
my $TIMEOUT = 5; # length of time, in seconds, until we 
		 # give up on a host
my @LDAP_HOSTS = ( "fake1.host.com",
	           "fake2.host.com" );
	           # The hosts which have LDAP running.
	           # Note that the first host listed is assumed
	           # to be the master
my $LDAP_CONF_PATH = "/etc/ldap.conf";
# end constant definitions-------------------------------------


# begin subroutine definitions---------------------------------

# tests to see if a host is alive
# takes the host name as a param
# returns 1 if alive, else 0
sub is_host_alive($) 
{
	my $host = shift();
	my $retval = 0;
	my $sock; # socket to test life

	$sock = new IO::Socket::INET(
		PeerAddr => $host,
		PeerPort => $LDAP_PORT,
		Proto => 'tcp',
		Timeout => $TIMEOUT
		);

	if ( $sock ) {
		$retval = 1;
		close( $sock );
	}

	return $retval;
}

# changes the URI line in the LDAP config file
# takes the whole URI line as a param
sub change_uri($) 
{
	my $uri = shift();
	my $CONF; # filehandle for config file
	my @lines; # the lines in the config file
	my $change = 0; # set to 1 if the uri line differs

	# try to open the file
	open( CONF, "<$LDAP_CONF_PATH" ) or die
		"Failed to open $LDAP_CONF_PATH for reading: $!";

	# read in the lines
	@lines = <CONF>;
	close( CONF );

	# find and change the uri line
	for( my $x = 0; $x <= $#lines; $x++ ) {
		if ( $lines[ $x ] =~ /^uri/ &&
		     $lines[ $x ] ne "$uri\n" ) {
			$change = 1;
			$lines[ $x ] = "$uri\n";
		}
	}

	# write out the config file, if needed
	if ( $change ) {
		open( CONF, ">$LDAP_CONF_PATH" ) or die
			"Failed to open $LDAP_CONF_PATH for writing: $!";
		print CONF @lines;
		close( CONF );
	}
}

# end subroutine definitions-----------------------------------


# main program code--------------------------------------------

my @alive_hosts; # hosts which respond
my $uri_line; # the new uri line

foreach my $current ( @LDAP_HOSTS ) {
	if ( is_host_alive( $current ) ) {
		push( @alive_hosts, $current );
	}
}

# make the uri line
$uri_line = "uri ";
foreach my $current ( @alive_hosts ) {
	$uri_line .= "$LDAP_PREFIX$current ";
}

# change the config file
change_uri( $uri_line );

# note that this must be run from cron or equivalent;
# it only runs once

Tinkster · 08-17-2008, 05:00 PM

Heh ... well, yet another alternative would then have been (assuming that your
clients use /etc/hosts and /etc/host.conf has the order files,bind) your script
could be changing the hosts file instead ...

linux=future · 06-21-2009, 10:20 AM

In case anyone is actually using / will use the posted script...

This morning, something ugly happened on our master LDAP server. It never actually went down; it could be pinged, and ports for provided services were still opened. However, those services, including LDAP, would never actually do anything. The above script only checks for the port, so it didn't catch that (much to the dismay of some users...)

I have posted below a revised version that actually checks to see if the server can respond or not. Again, no warranties, redistribute if you want, etc. It requires that Net::LDAPS is installed.

Code:

#!/usr/bin/perl -w

# This script updates /etc/ldap.conf based upon which
# LDAP servers are up or not.  This is so utilities like
# finger do not query the LDAP servers which are down

# Note that the is_host_alive routine is based upon
# De Bodt Lieven's code from 
# http://www.monitoringexchange.org/cgi-bin/page.cgi?g=Detailed%2F1325.html;d=1

use strict;
use Net::LDAPS;

# define some constants ---------------------------------------
my $LDAP_PORT = 636; # port used by LDAP; I use secure LDAP
my $LDAP_VERSION = 3; # version of LDAP to test for
my $LDAP_PREFIX = "ldaps://"; # prefix for an LDAP URI
	                      # use ldap:// for normal LDAP
my $TIMEOUT = 5; # length of time, in seconds, until we 
		 # give up on a host
my @LDAP_HOSTS = ( "fake1.host.com",
	           "fake2.host.com" );
	           # The hosts which have LDAP running.
	           # Note that the first host listed is assumed
	           # to be the master
my $LDAP_CONF_PATH = "/etc/ldap.conf";
# end constant definitions-------------------------------------


# begin subroutine definitions---------------------------------

# tests to see if a host is alive
# takes the host name as a param
# returns 1 if alive, else 0
sub is_host_alive($) 
{
	my $host = shift();
	my $retval = 0;
	my $ldap; # session with the LDAP server

	# create the session
	$ldap = Net::LDAPS->new( $host, 
				 port => $LDAP_PORT,
				 version => 3,
				 timeout => $TIMEOUT );
	if( $ldap ) {
		my $resp; # response from LDAP server
		$resp = $ldap->bind();
		unless( $resp->code ) {
			# connection successful
			$resp = $ldap->unbind(); #disconnect
			$retval = 1;
		}
	}

	return $retval;
}

# changes the URI line in the LDAP config file
# takes the whole URI line as a param
sub change_uri($) 
{
	my $uri = shift();
	my $CONF; # filehandle for config file
	my @lines; # the lines in the config file
	my $change = 0; # set to 1 if the uri line differs

	# try to open the file
	open( CONF, "<$LDAP_CONF_PATH" ) or die
		"Failed to open $LDAP_CONF_PATH for reading: $!";

	# read in the lines
	@lines = <CONF>;
	close( CONF );

	# find and change the uri line
	for( my $x = 0; $x <= $#lines; $x++ ) {
		if ( $lines[ $x ] =~ /^uri/ &&
		     $lines[ $x ] ne "$uri\n" ) {
			$change = 1;
			$lines[ $x ] = "$uri\n";
		}
	}

	# write out the config file, if needed
	if ( $change ) {
		open( CONF, ">$LDAP_CONF_PATH" ) or die
			"Failed to open $LDAP_CONF_PATH for writing: $!";
		print CONF @lines;
		close( CONF );
	}
}

# end subroutine definitions-----------------------------------


# main program code--------------------------------------------

my @alive_hosts; # hosts which respond
my $uri_line; # the new uri line

foreach my $current ( @LDAP_HOSTS ) {
	if ( is_host_alive( $current ) ) {
		push( @alive_hosts, $current );
	}
}

# make the uri line
$uri_line = "uri ";
foreach my $current ( @alive_hosts ) {
	$uri_line .= "$LDAP_PREFIX$current ";
}

# change the config file
change_uri( $uri_line );

# note that this must be run from cron or equivalent;
# it only runs once

mrlondon · 03-31-2010, 10:27 AM

Our main ldap server has hung twice in the last few weeks (a Windows server). Nothing in the logs shows that anything was wrong with it. It was pingable, but couldn't be connected to. Did you ever find the reason why your ldap server was hanging? Thanks. - Mark