named giving me lot of trouble in centos 6.5

Red Squirrel · 10-26-2014, 08:53 PM

I'm working on migrating to a new server and named is giving me tons of issues:

1: At random, the /var/named/chroot/etc/named.conf file VANISHES! WTF? Typically this happens when it fails to load. Then I have to restore from backup. Why is it doing this? Also when I go to vim it from another folder it wants to create a directory. I have to cd to that folder to edit it (that's when it does not vanish). There seems to be a backup in /etc/ thankfully. Overall it's acting very messed up.

2: Every time I go to restart named, I get this error:

Code:

umount: /var/named/chroot/var/named: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

3: On that note, another thing I noticed is when I do df I get lot of named related garbage:

Code:

Filesystem      Size  Used Avail Use% Mounted on
rootfs          1.9T  296G  1.5T  17% /
/dev/root       1.9T  296G  1.5T  17% /
devtmpfs         16G  264K   16G   1% /dev
tmpfs            16G     0   16G   0% /dev/shm
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/etc/named
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/var/named
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/etc/named.rfc1912.zones
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/etc/rndc.key
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/usr/lib64/bind
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/etc/named.iscdlv.key
/dev/root       1.9T  296G  1.5T  17% /var/named/chroot/etc/named.root.key

What in the world is all that crap? Why is it creating all these weird mount points? Is there a way to get rid of this? Kind of annoying.

Overall this version of named seems really flaky. Is there a way to fix all of this?

Red Squirrel · 10-26-2014, 10:06 PM

Another thing I just realized is it only works locally, it refuses to resolve anything from an external server.

Ex: if I use nslookup this is what I get:

Code:

> server 192.99.10.155
Default server: 192.99.10.155
Address: 192.99.10.155#53
> testdomain.loc
Server:		192.99.10.155
Address:	192.99.10.155#53

** server can't find testdomain.loc.loc: REFUSED
> iceteks.com
Server:		192.99.10.155
Address:	192.99.10.155#53

** server can't find iceteks.com.loc: REFUSED
> iceteks.com.
Server:		192.99.10.155
Address:	192.99.10.155#53

** server can't find iceteks.com: REFUSED
>

#1: it keeps adding .loc after. No idea why. But even without that, it still does not work. This is the contents of /var/named/chroot/etc/named.conf:

Code:

options {
        listen-on port 53 { 127.0.0.1; 192.99.10.155; 192.99.10.156;};
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query     { localhost; };
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/var/named/zones.conf";

Mostly everything is default except for zone.conf file and the actual zone files which are verbatim from the working DNS server. I ensured to open up the DNS port in the firewall.

Red Squirrel · 10-26-2014, 11:58 PM

It *seems* to work now, but I'll have to leave it alone overnight and see.

zhjim · 10-27-2014, 03:21 AM

The output of mount gives me trouble. Specially cause the root file system seems to be mounted on the named working directory. I would say this is also the reason for the named.conf to disappear. Nother thing I can't get my head around is why the restart of named unmounts a filesystem?

How did you install named in the first place?

Red Squirrel · 10-27-2014, 12:45 PM

Quote:

Originally Posted by zhjim

The output of mount gives me trouble. Specially cause the root file system seems to be mounted on the named working directory. I would say this is also the reason for the named.conf to disappear. Nother thing I can't get my head around is why the restart of named unmounts a filesystem?

How did you install named in the first place?

All I did is "yum install named" nothing special. This is a dedicated server so the OS was preloaded by OVH (the provider).

Another weird thing I noticed is there is another chroot folder within the /var/named/chroot/var/named folder. This is a mess.

For all intent and purposes it seems to be working now though... but still find it weird how it's acting so differently than any other named installation I've seen. All those fake mount points are also kind of a pain in the ass. Seems like really dirty programming, whatever they're trying to accomplish with all this.

Red Squirrel · 10-27-2014, 04:43 PM

ok I take that back, this version of named is flacky as hell. Suddenly it's not working anymore. I was right in the middle of transferring some of my domains to the new server then realized none of them work. This is a mess.

Just getting SURVFAIL left and right when I use nslookup and all my sites are down. Part of the issue might be that my new server provider does not provide more than 1 IP without having to order more and they're being slow at provisioning them so I have ns1 and ns2 pointed to the same IP, could that be an issue?

edit: And now it works... Now I transffered everything over to the new nameserver so I think I need to stop touching it for a while and see what happens... I think this issue is a combination of my browser constantly wanting to cache stuff and my local DNS also caching stuff. If I use isup.me everything does work.

zhjim · 10-28-2014, 03:04 AM

I don't think that pointing the two ns record to the same IP would cause such a behaviour. Also its not the intended way to use nameservers it should be okay for the start. Just get a second server and point the other ns record this one. Two IP's on the same server don't bring redundence into your nameserver setup.

The install way seems valid for me. Also long time ago i fidled with rpm based distros.
How do you restart the server? /etc/rc.d/named restart? or similar? If so check on the script and see if you find any mount commands inside. Which just should not be there in my opinion.

Maybe you should see if you can setup a fresh machine at home and rebuild your server. If this works out well, maybe just reinstall the server with the new build image. At least you would know if all of thats what happening is "normal" behaviour. Which I doubt.

Red Squirrel · 10-28-2014, 12:28 PM

This is a dedicated server that is online facing, so it cost a lot per month, not getting a second box. I don't really have the option to host from home (ISP wont allow it) otherwise I would be hosting everything from home, it would be much easier!

This is the entire init file for named:

Code:

#!/bin/bash
#
# named           This shell script takes care of starting and stopping
#                 named (BIND DNS server).
#
# chkconfig: - 13 87
# description: named (BIND) is a Domain Name Server (DNS) \
# that is used to resolve host names to IP addresses.
# probe: true

### BEGIN INIT INFO
# Provides: $named
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $network $syslog
# Default-Start:
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: start|stop|status|restart|try-restart|reload|force-reload DNS server
# Description: control ISC BIND implementation of DNS server
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

[ -r /etc/sysconfig/named ] && . /etc/sysconfig/named

RETVAL=0
export KRB5_KTNAME=${KEYTAB_FILE:-/etc/named.keytab}

named='named'
if [ -x /usr/sbin/named-sdb ]; then
        named='named-sdb'
fi

# Don't kill named during clean-up
NAMED_SHUTDOWN_TIMEOUT=${NAMED_SHUTDOWN_TIMEOUT:-25}

if [ -n "$ROOTDIR" ]; then
   ROOTDIR=`echo $ROOTDIR | sed 's#//*#/#g;s#/$##'`;
   rdl=`/usr/bin/readlink $ROOTDIR`;
   if [ -n "$rdl" ]; then
      ROOTDIR="$rdl";
   fi;
fi

PIDFILE="/var/run/named/named.pid"

ROOTDIR_MOUNT='/etc/named /etc/pki/dnssec-keys /var/named /etc/named.conf
/etc/named.dnssec.keys /etc/named.rfc1912.zones /etc/rndc.conf /etc/rndc.key
/usr/lib64/bind /usr/lib/bind /etc/named.iscdlv.key /etc/named.root.key'

mount_chroot_conf()
{
  if [ -n "$ROOTDIR" ]; then
    for all in $ROOTDIR_MOUNT; do
      # Skip nonexistant files
      [ -e "$all" ] || continue

      # If mount source is a file
      if ! [ -d "$all" ]; then
        # mount it only if it is not present in chroot or it is empty
        if ! [ -e "$ROOTDIR$all" ] || [ `stat -c'%s' "$ROOTDIR$all"` -eq 0 ]; then
          touch "$ROOTDIR$all"
          mount --bind "$all" "$ROOTDIR$all"
        fi
      else
        # Mount source is a directory. Mount it only if directory in chroot is
        # empty.
        if [ -e "$all" ] && [ `ls -1A $ROOTDIR$all | wc -l` -eq 0 ]; then
          mount --bind "$all" "$ROOTDIR$all"
        fi
      fi
    done
  fi
}

umount_chroot_conf()
{
  if [ -n "$ROOTDIR" ]; then
    for all in $ROOTDIR_MOUNT; do
      # Check if file is mount target. Do not use /proc/mounts because detecting
      # of modified mounted files can fail.
      if mount | grep -q '.* on '"$ROOTDIR$all"' .*'; then
        umount "$ROOTDIR$all"
        # Remove temporary created files
        [ -f "$all" ] && rm -f "$ROOTDIR$all"
      fi
    done
  fi
}

pidofnamed() {
        pidofproc -p "$ROOTDIR/$PIDFILE" "$named";
}

# Check if all what named needs running
start()
{
  [ "$EUID" != "0" ] && exit 4

  # Source networking configuration.
  [ -r /etc/sysconfig/network ] && . /etc/sysconfig/network

  # Check that networking is up
  [ "${NETWORKING}" = "no" ] && exit 1


  [ -x /usr/sbin/"$named" ] || exit 5

  if [ ! -s /etc/rndc.key ]; then
    # Generate rndc.key if doesn't exist
    echo -n $"Generating /etc/rndc.key:"
    if /usr/sbin/rndc-confgen -a > /dev/null 2>&1; then
      chmod 640 /etc/rndc.key
      chown root.named /etc/rndc.key
      [ -x /sbin/restorecon ] && /sbin/restorecon /etc/rndc.key
      success $"/etc/rndc.key generation"
      echo
    else
      failure $"/etc/rndc.key generation"
      echo
    fi
  fi

  # Handle -c option
  previous_option='unspecified';
  for a in $OPTIONS; do
    if [ $previous_option = '-c' ]; then
      named_conf=$a;
    fi;
    previous_option=$a;
  done;

  named_conf=${named_conf:-/etc/named.conf};

  mount_chroot_conf

  if [ ! -r $ROOTDIR$named_conf ]; then
    echo 'Cannot find configuration file. You could create it by system-config-bind'
    exit 6;
  fi;

  [ -x /sbin/portrelease ] && /sbin/portrelease named &>/dev/null || :

  # all pre-start is done, lets start named
  echo -n $"Starting named: "
  if [ -n "`pidofnamed`" ]; then
    echo -n $"named: already running"
    success
    echo
    exit 0;
  fi;

  if ! [ "$DISABLE_ZONE_CHECKING" = yes ]; then
    ckcf_options='-z'; # enable named-checkzone for each zone (9.3.1+) !
  fi;

  if [ -n "${ROOTDIR}" -a "x${ROOTDIR}" != "x/" ]; then
    OPTIONS="${OPTIONS} -t ${ROOTDIR}"
    ckcf_options="$ckcf_options -t ${ROOTDIR}";
    [ -s /etc/localtime ] && cp -fp /etc/localtime ${ROOTDIR}/etc/localtime;
  fi

  RETVAL=0
  # check if configuration is correct
  if [ -x /usr/sbin/named-checkconf ] && [ -x /usr/sbin/named-checkzone ] && /usr/sbin/named-checkconf $ckcf_options ${named_conf} >/dev/null 2>&1; then

    daemon --pidfile "$ROOTDIR/$PIDFILE" /usr/sbin/"$named" -u named ${OPTIONS};
    RETVAL=$?
    if [ $RETVAL -eq 0 ]; then
      rm -f /var/run/{named,named-sdb}.pid;
      ln -s $ROOTDIR/"$PIDFILE" /var/run/"$named".pid;
    fi;

  else
    named_err="`/usr/sbin/named-checkconf $ckcf_options $named_conf 2>&1`";        
    echo
    echo "Error in named configuration:";
    echo "$named_err";
    failure
    echo
    [ -x /usr/bin/logger ] && echo "$named_err" | /usr/bin/logger -pdaemon.error -tnamed;
    umount_chroot_conf
    exit 2;
  fi;
  echo
  if [ $RETVAL -eq 0 ]; then
    touch /var/lock/subsys/named;
  else
    umount_chroot_conf
    exit 7;
  fi
  return 0;
}

stop() {
  [ "$EUID" != "0" ] && exit 4

  # Stop daemons.
  echo -n $"Stopping named: "
  [ -x /usr/sbin/rndc ] && /usr/sbin/rndc stop >/dev/null 2>&1;
  RETVAL=$?
  # was rndc successful?
  [ "$RETVAL" -eq 0 ] || \
    killproc -p "$ROOTDIR/$PIDFILE" "$named" -TERM >/dev/null 2>&1

  timeout=0
  RETVAL=0
  while pidofnamed &>/dev/null; do
    if [ $timeout -ge $NAMED_SHUTDOWN_TIMEOUT ]; then
      RETVAL=1
      break
    else
      sleep 2 && echo -n "."
      timeout=$((timeout+2))
    fi;
  done

  umount_chroot_conf

  # remove pid files
  if [ $RETVAL -eq 0 ]; then
    rm -f /var/lock/subsys/named
    rm -f /var/run/{named,named-sdb}.pid
  fi;

  if [ $RETVAL -eq 0 ]; then
    success
  else
    failure
    RETVAL=1
  fi;
  echo
  return $RETVAL
}


rhstatus() {
  [ -x /usr/sbin/rndc ] && /usr/sbin/rndc status;
  status -p "$ROOTDIR/$PIDFILE" -l named /usr/sbin/"$named";
  return $?
}
restart() {
        stop
        start
}
reload() {
        [ "$EUID" != "0" ] && exit 

        echo -n $"Reloading "$named": "
        p=`pidofnamed`
        RETVAL=$?
        if [ "$RETVAL" -eq 0 ]; then 
            /usr/sbin/rndc reload >/dev/null 2>&1 || /bin/kill -HUP $p;
            RETVAL=$?
        fi
        [ "$RETVAL" -eq 0 ] && success $"$named reload" || failure $"$named reload"
        echo
        return $RETVAL
}

checkconfig() {
        ckcf_options='-z';
        if [ -n "${ROOTDIR}" -a "x${ROOTDIR}" != "x/" ]; then
                ckcf_options="$ckcf_options -t ${ROOTDIR}";
                mount_chroot_conf
        fi;
        if [ -x /usr/sbin/named-checkconf ] && [ -x /usr/sbin/named-checkzone ] && /usr/sbin/named-checkconf $ckcf_options ${named_conf} ; then
                umount_chroot_conf
                return 0;
        else
                umount_chroot_conf
                return 1;
        fi
}

# See how we were called.
case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        status)
                rhstatus;
                RETVAL=$?
                ;;
        restart)
                restart
                ;;
        condrestart|try-restart)
                if [ -e /var/lock/subsys/named ]; then restart; fi
                ;;
        reload)
                reload
                ;;
        force-reload)
                if ! reload; then restart; fi
                ;;
        checkconfig|configtest|check|test)
                checkconfig
                ;;
        *)
                echo $"Usage: $0 {start|stop|status|restart|try-restart|reload|force-reload}"
                [ "x$1" = "x" ] && exit 0
                exit 2
esac

exit $RETVAL

There does seem to be some mount stuff in there. No idea why named would need to mount anything or do anything having to do with the file system.