Slackware 14.2 server

orbea · 11-09-2018, 09:35 AM

I have a Slackware64 14.2 server instance hosted by kimsufi, I updated to the 4.4.157 kernel from an older 4.4.* kernel (4.4.132? I accidentally installed over it, don't do that!) and now the server will boot without any network. Any console access without a network is also not available and that makes it hard to debug.

However the kimsufi network kernels still work where i have tested 4.14.63 and 4.9.120. I next tried the Slackware current 4.19.1 kernel, my own compiled 4.14.79 and the original 14.2 4.4.14 kernel with the same non-working results.

After reading /var/log/messages the issue seems to be in rc.inet1 in the function if_up.

The working kimsufi 4.14.63 kernel. (I changed the IP addresses)

Code:

Nov  8 15:04:56 ns root: /etc/rc.d/rc.inet1:  /sbin/ifconfig lo 127.0.0.1
Nov  8 15:04:56 ns root: /etc/rc.d/rc.inet1:  /sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo
Nov  8 15:04:56 ns root: /etc/rc.d/rc.inet1:  /sbin/ifconfig eth0 ***.***.***.** broadcast ***.***.***.*** netmask 255.255.255.0
Nov  8 15:04:57 ns kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov  8 15:04:57 ns root: /etc/rc.d/rc.inet1:  /sbin/route add default gw ***.***.***.*** metric 1
Nov  8 15:04:57 ns kernel: random: ssh-keygen: uninitialized urandom read (32 bytes read)
Nov  8 15:04:57 ns kernel: random: sshd: uninitialized urandom read (32 bytes read)

Slackware kernels

Code:

Nov  8 14:58:49 nsroot: /etc/rc.d/rc.inet1:  /sbin/ifconfig lo 127.0.0.1
Nov  8 14:58:49 ns root: /etc/rc.d/rc.inet1:  /sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo
Nov  8 14:58:50 ns root: /etc/rc.d/rc.inet1:  /sbin/route add default gw ***.***.***.*** metric 1
Nov  8 14:58:50 ns root: SIOCADDRT: Network is unreachable
Nov  8 14:58:50 ns kernel: [   12.450102] NET: Registered protocol family 10

I next edited the if_up function with more logging output.

Code:

# Function to bring up a network interface.  If the interface is
# already up or does not yet exist (perhaps because the kernel driver
# is not loaded yet), do nothing.
if_up() {
echo "INET1 DEBUG: Entering if_up()" | $LOGGER
if [ -e /proc/net/dev ]; then
echo "INET1 DEBUG: /proc/net/dev exists" | $LOGGER
else
echo "INET1 DEBUG: /proc/net/dev not found" | $LOGGER
fi
echo "INET1 DEBUG: /sys/class/net/${1%%:*}" | $LOGGER
if [ -e /sys/class/net ]; then
echo "INET1 DEBUG: /sys/class/net exists" | $LOGGER
[ -e /sys/class/net/${1%%:*} ] && echo "INET1 DEBUG: /sys/class/net/${1%%:*} exists" | $LOGGER
else
echo "INET1 DEBUG: /sys/class/net not found" | $LOGGER
fi
  # Determine position 'i' of this interface in the IFNAME array:
  i=0
  while [ $i -lt $MAXNICS ]; do
    [ "${IFNAME[$i]}" = "${1}" ] && break
    i=$(($i+1))
  done
  # If the interface is a bridge, then create it first:
  [ -n "${BRNICS[$i]}" ] && br_open $i
  # If the interface isn't in the kernel yet (but there's an alias for it in
  # modules.conf), then it should be loaded first:
  if ! grep `echo ${1}: | cut -f 1 -d :`: /proc/net/dev 1> /dev/null ; then # no interface yet
echo "INET1 DEBUG: 1" | $LOGGER
    if /sbin/modprobe -c | grep -v "^#" | grep -w "alias ${1}" | grep -vw "alias ${1} off" > /dev/null ; then
echo "INET1 DEBUG: 2" | $LOGGER
      echo "/etc/rc.d/rc.inet1:  /sbin/modprobe ${1}" | $LOGGER
      /sbin/modprobe ${1}
    fi
  fi
  if grep `echo ${1}: | cut -f 1 -d :`: /proc/net/dev 1> /dev/null ; then # interface exists
echo "INET1 DEBUG: 3" | $LOGGER
    if ! /sbin/ifconfig | grep -w "${1}" 1>/dev/null || \n      ! /sbin/ifconfig ${1} | grep -w inet 1> /dev/null ; then # interface not up or not configured
echo "INET1 DEBUG: 4" | $LOGGER
      if [ ! "${HWADDR[$i]}" = "" ]; then # Set hardware address _before_ the interface goes up:
echo "INET1 DEBUG: 5" | $LOGGER
        echo "/etc/rc.d/rc.inet1:  /sbin/ifconfig ${1} hw ether ${HWADDR[$i]}" | $LOGGER
        /sbin/ifconfig ${1} hw ether ${HWADDR[$i]}
      fi
      if [ ! "${MTU[$i]}" = "" ]; then # Set MTU to something else than 1500
echo "INET1 DEBUG: 6" | $LOGGER
        echo "/etc/rc.d/rc.inet1:  /sbin/ifconfig ${1} mtu ${MTU[$i]}" | $LOGGER
        /sbin/ifconfig ${1} mtu ${MTU[$i]}
      fi
      if [ -x /etc/rc.d/rc.wireless ]; then
echo "INET1 DEBUG: 7" | $LOGGER
        . /etc/rc.d/rc.wireless ${1} start # Initialize any wireless parameters
      fi
      if [ "${USE_DHCP[$i]}" = "yes" ]; then # use DHCP to bring interface up
echo "INET1 DEBUG: 8" | $LOGGER
        # Clear DHCP_OPTIONS before adding new options to it:
        unset DHCP_OPTIONS
        # Set DHCP_OPTIONS for this interface:
        [ ${DHCP_HOSTNAME[$i]} ] && DHCP_OPTIONS="-h ${DHCP_HOSTNAME[$i]}"
        [ "${DHCP_KEEPRESOLV[$i]}" = "yes" ] && DHCP_OPTIONS="$DHCP_OPTIONS -C resolv.conf"
        [ "${DHCP_KEEPNTP[$i]}" = "yes" ] && DHCP_OPTIONS="$DHCP_OPTIONS -C ntp.conf"
        [ "${DHCP_KEEPGW[$i]}" = "yes" ] && DHCP_OPTIONS="$DHCP_OPTIONS -G"
        [ "${DHCP_DEBUG[$i]}" = "yes" ] && DHCP_OPTIONS="$DHCP_OPTIONS -d"
        [ "${DHCP_NOIPV4LL[$i]}" = "yes" ] && DHCP_OPTIONS="$DHCP_OPTIONS -L"
        [ ${DHCP_IPADDR[$i]} ] && DHCP_OPTIONS="$DHCP_OPTIONS -r ${DHCP_IPADDR[$i]}"
        echo "Polling for DHCP server on interface ${1}:"
        # If you set a timeout, you get one, even if the kernel doesn't think that
        # your device is connected, in case /sys isn't right (which it usually isn't
        # except right after the device is loaded, when it usually is):
        #### (start commented out)
        # This is deactivated for now since the kernel has been returning incorrect
        # results concerning whether the interface carrier is detected.
        #if [ "${DHCP_TIMEOUT[$i]}" = "" ]; then
        #  ifconfig ${1} up && sleep 1
        #  CONNSTATUS="$(cat /sys/class/net/${1}/carrier 2> /dev/null)"
        #  ifconfig ${1} down
        #  if [ "$CONNSTATUS" = "0" ]; then
        #    # The kernel has just told us the cable isn't even plugged in, but we will
        #    # give any DHCP server a short chance to reply anyway:
        #    echo "No carrier detected on ${1}.  Reducing DHCP timeout to 10 seconds."
        #    DHCP_TIMEOUT[$i]=10
        #  fi
        #fi
        #### (end commented out)
        # 10 seconds should be a reasonable default DHCP timeout.  30 was too much.
        echo "/etc/rc.d/rc.inet1:  /sbin/dhcpcd -t ${DHCP_TIMEOUT[$i]:-10} ${DHCP_OPTIONS} ${1}" | $LOGGER
        /sbin/dhcpcd -t ${DHCP_TIMEOUT[$i]:-10} ${DHCP_OPTIONS} ${1}
      else # bring up interface using a static IP address
echo "INET1 DEBUG: 9" | $LOGGER
        if [ ! "${IPADDR[$i]}" = "" ]; then # skip unconfigured interfaces
echo "INET1 DEBUG: 10" | $LOGGER
          # Determine broadcast address from the IP address and netmask:
          BROADCAST[$i]=`/bin/ipmask ${NETMASK[$i]} ${IPADDR[$i]}|cut -f1 -d' '`
          # Set up the network card:
          echo "/etc/rc.d/rc.inet1:  /sbin/ifconfig ${1} ${IPADDR[$i]} broadcast ${BROADCAST[$i]} netmask ${NETMASK[$i]}" | $LOGGER
          /sbin/ifconfig ${1} ${IPADDR[$i]} broadcast ${BROADCAST[$i]} netmask ${NETMASK[$i]}
        else

echo "INET1 DEBUG: 11" | $LOGGER
          if [ "$DEBUG_ETH_UP" = "yes" ]; then
echo "INET1 DEBUG: 12" | $LOGGER
            echo "/etc/rc.d/rc.inet1:  ${1} interface is not configured in /etc/rc.d/rc.inet1.conf" | $LOGGER
          fi
        fi
      fi
    else
echo "INET1 DEBUG: 13" | $LOGGER
      if [ "$DEBUG_ETH_UP" = "yes" ]; then
echo "INET1 DEBUG: 14" | $LOGGER
        echo "/etc/rc.d/rc.inet1:  ${1} is already up, skipping" | $LOGGER
      fi
    fi 
  else
echo "INET1 DEBUG: 14" | $LOGGER
    if [ "$DEBUG_ETH_UP" = "yes" ]; then
echo "INET1 DEBUG: 15" | $LOGGER
      echo "/etc/rc.d/rc.inet1:  ${1} interface does not exist (yet)" | $LOGGER
    fi
  fi
}

This made it more clear what its doing different.

With the kimsufi kernels.

Code:

Nov  9 01:00:10 ns root: /etc/rc.d/rc.inet1:  /sbin/ifconfig lo 127.0.0.1
Nov  9 01:00:10 nsroot: /etc/rc.d/rc.inet1:  /sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth0
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth0 exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 3
Nov  9 01:00:10 ns root: INET1 DEBUG: 4
Nov  9 01:00:10 ns root: INET1 DEBUG: 9
Nov  9 01:00:10 ns root: INET1 DEBUG: 10
Nov  9 01:00:10 ns root: /etc/rc.d/rc.inet1:  /sbin/ifconfig eth0 ***.***.***.** broadcast ***.***.***.*** netmask 255.255.255.0
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth1
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 1
Nov  9 01:00:10 ns root: INET1 DEBUG: 14
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth2
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 1
Nov  9 01:00:10 ns root: INET1 DEBUG: 14
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth3
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 1
Nov  9 01:00:10 ns root: INET1 DEBUG: 14
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth4
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 1
Nov  9 01:00:10 ns root: INET1 DEBUG: 14
Nov  9 01:00:10 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:00:10 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net/eth5
Nov  9 01:00:10 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:00:10 ns root: INET1 DEBUG: 1
Nov  9 01:00:10 ns root: INET1 DEBUG: 14
Nov  9 01:00:10 ns root: /etc/rc.d/rc.inet1:  /sbin/route add default gw ***.***.***.*** metric 1
Nov  9 01:00:10 ns kernel: random: ssh-keygen: uninitialized urandom read (32 bytes read)
Nov  9 01:00:10 ns kernel: random: sshd: uninitialized urandom read (32 bytes read)

Slackware kernels.

Code:

Nov 9 01:05:15 ns root: /etc/rc.d/rc.inet1:  /sbin/ifconfig lo 127.0.0.1
Nov  9 01:05:15 ns root: /etc/rc.d/rc.inet1:  /sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth0
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth1
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth2
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth3
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth4
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: INET1 DEBUG: Entering if_up()
Nov  9 01:05:15 ns root: INET1 DEBUG: /proc/net/dev exists
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net/eth5
Nov  9 01:05:15 ns root: INET1 DEBUG: /sys/class/net exists
Nov  9 01:05:15 ns root: INET1 DEBUG: 1
Nov  9 01:05:15 ns root: INET1 DEBUG: 14
Nov  9 01:05:15 ns root: /etc/rc.d/rc.inet1:  /sbin/route add default gw ***.***.***.*** metric 1
Nov  9 01:05:15 ns root: SIOCADDRT: Network is unreachable
Nov  9 01:05:15 ns kernel: [   12.358433] NET: Registered protocol family 10
Nov  9 01:05:15 ns kernel: [   12.376920] Segment Routing with IPv6

This makes me think the issue may be that the following line is never true.

Code:

if grep `echo ${1}: | cut -f 1 -d :`: /proc/net/dev 1> /dev/null ; then # interface exists

The script in current is a little different and some of the changes may help, but as every failed test results in having to hard reboot it via the kimsufi control panel I would like to understand what is going wrong more.

Any ideas what is happening or how I can get more helpful information?

Hangaber · 11-09-2018, 11:55 AM

(Edit out your hostname. Mine can be used to figure out the IP.)

Try a slack huge kernel to test, or compile another kernel and make sure the modules are good. My eth0 uses module: e1000e, but yours is possibly different.

FYI my Kimsufi box does show the "ifconfig eth0" line.

Code:

Oct  9 22:33:36 myhostname root: /etc/rc.d/rc.inet1:  /sbin/ifconfig eth0 ***.***.***.*** broadcast ***.***.***.255 netmask 255.255.255.0
Oct  9 22:33:38 myhostname root: /etc/rc.d/rc.inet1:  /sbin/route add default gw ***.***.***.*** metric 1

Code:

#cat /etc/slackware-version 
Slackware 14.2
# uname -r
4.4.159

orbea · 11-09-2018, 12:09 PM

I forgot to mention that I tried both huge and generic kernels, I can try building more kernels, but do you have any specific versions you would suggest? Just guessing doesn't seem very productive...

Hangaber · 11-09-2018, 01:54 PM

Primarily giving you hope that 4.4.157 in slack on kimsufi should have worked, and I think the key here is possibly the kernel or modules or boot params - and not directly the slack inet1 script.

If I was in the same situation, I'd just go thru the kernel re-compile process, re-compile modules, new initrd, update grub. I pretty much use the 4.4.* generic config with it's "make oldconfig" changes.

Just throwing some things out there to check/try perhaps;

Compare boot params between your kimsufi and slack boot (`cat /proc/cmdline`)

What's the module that the kimsufi kernel is using for your network interface? Is it loaded in your no-network instance, was the module compiled (in kernel or as a loadable module)? (Mine is "CONFIG_E1000E=m", and `make modules_install` put it at "/lib/modules/4.4.159/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko")

If the module is already loaded, and eth0 exists but just isn't configured, can you ifconfig and route add it manually (well, via script in your case)..

Code:

/sbin/ifconfig eth0 YOUR_IP broadcast BROADCAST_ADDR netmask 255.255.255.0
/sbin/route add default gw GATEWAY_IP metric 1

In my config, the gateway is similar to my IP, ending in .254, and the broadcast addy is .255. I'm assuming your netmask is the same 255.255.255.0

Also, no ipv6 for me.

abga · 11-09-2018, 02:34 PM

@orbea

It looks like your ethernet adapter(s) is not properly initialized (kernel/driver) and I'm not sure I understand what's your actual level of access to that server instance.
If you're able to open a (virtual)console and run some commands, I'd suggest to look in dmesg after how/if the ethernet adapter was initialized on driver level instead of focusing on rc.inet1. There are some other investigative steps that can be done if you are able to obtain a bash prompt on the running system.
Updating the PCI ID file with "/usr/sbin/update-pciids" might also help with your new kernel.
You can temporarily disable/rename /etc/rc.d/rc.inet1 - remove the exec bit, and put your simple manual networking setup in the new /etc/rc.d/rc.inet1 or in rc.local (if you're not starting some services in /etc/rc.d/rc.inet2 that require the networking up & running):

Code:

#!/bin/bash
#setup lo
/sbin/ifconfig lo 127.0.0.1
/sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo
#setup eth0
/sbin/ifconfig eth0 ***.***.***.** broadcast ***.***.***.*** netmask 255.255.255.0
/sbin/route add default gw ***.***.***.***

I also don't know how you upgraded the kernel, but for the next upgrade I'd suggest to clean the older /lib/modules/x.x & /lib/firmware and make sure your /boot folder contains the proper files.

hotchili · 11-09-2018, 08:44 PM

I have a server at soyoustart running 14.2, which is a brand like kimsufi from OVH so the setups should be quite similar.
For me I always use the provided kernel config as base and then build my own, I just change the localname
and a few other config options which I like. They have everything baked in and module loading disabled, with that
it's just dropping the bzImage and System.map into /boot and rerun grub-mkconfig -o /boot/grub/grub.cfg

You can find the current OVH standard kernels and config, for example for 4.9.135 here:
https://last-public-ovh-kernel.snap....x-std-ipv6-64/

they used to be located here: ftp://ftp.ovh.net/made-in-ovh/bzImage/ but seems they have moved.

Those kernels are very generic, so I think they work on all OVH servers, maybe this is an option.