CentOS 7 Mystery: Network does not start on boot, but will start manually

deathsfriend99 · 02-09-2021, 11:23 AM

I've got a mystery.
I have a few CentOS 7 boxes that the network will not start on boot, but after boot, I can manually start it just fine.
First, these are lean boxes with no network manager.
They are also clones.

chkconfig network.service on

Here's the error:

Code:

● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2021-02-09 10:49:41 EST; 1h 21min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1613 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)

Feb 09 10:49:40 hostname systemd[1]: Starting LSB: Bring up/down networking...
Feb 09 10:49:40 hostname network[1613]: Bringing up loopback interface:  [  OK  ]
Feb 09 10:49:41 hostname network[1613]: Bringing up interface em4:  ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device em4 does not seem to be present, delaying initialization.
Feb 09 10:49:41 hostname network[1613]: [FAILED]
Feb 09 10:49:41 hostname systemd[1]: network.service: control process exited, code=exited status=1
Feb 09 10:49:41 hostname systemd[1]: Failed to start LSB: Bring up/down networking.
Feb 09 10:49:41 hostname systemd[1]: Unit network.service entered failed state.
Feb 09 10:49:41 hostname systemd[1]: network.service failed.

But if I start it manually it works fine.

Code:

[root@hostname ~]# systemctl start network
[ 5203.845914 ] igb 000:06:00.1 em4: igb: em4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

/etc/sysconfig/network:

Code:

NETWORKING=yes

/etc/sysconfig/networking-scripts/ifcg-em4:

Code:

DEVICE=em4
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.x.x.x
NETMASK=255.255.255.0
HWADDR=x:x:x:x
NM_CONTROLLED=no

I have checked /etc/udev/rules.d/70-persistent-net.rules, there is no such file.

lazydog · 02-09-2021, 04:18 PM

What does the following commnad show:

Code:

sudo systemctl list-unit-files --type=service | grep Network

deathsfriend99 · 02-10-2021, 08:07 AM

That command returns nothing, but...

Code:

[root@hostname ~]# chkconfig --list |grep network

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

network        	0:off	1:off	2:on	3:on	4:on	5:on	6:off

TB0ne · 02-10-2021, 08:42 AM

Quote:

Originally Posted by deathsfriend99

I've got a mystery.
I have a few CentOS 7 boxes that the network will not start on boot, but after boot, I can manually start it just fine. First, these are lean boxes with no network manager. They are also clones.

chkconfig network.service on

Here's the error:

Code:

● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2021-02-09 10:49:41 EST; 1h 21min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1613 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)

Feb 09 10:49:40 hostname systemd[1]: Starting LSB: Bring up/down networking...
Feb 09 10:49:40 hostname network[1613]: Bringing up loopback interface:  [  OK  ]
Feb 09 10:49:41 hostname network[1613]: Bringing up interface em4:  ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device em4 does not seem to be present, delaying initialization.
Feb 09 10:49:41 hostname network[1613]: [FAILED]
Feb 09 10:49:41 hostname systemd[1]: network.service: control process exited, code=exited status=1
Feb 09 10:49:41 hostname systemd[1]: Failed to start LSB: Bring up/down networking.
Feb 09 10:49:41 hostname systemd[1]: Unit network.service entered failed state.
Feb 09 10:49:41 hostname systemd[1]: network.service failed.

But if I start it manually it works fine.

Code:

[root@hostname ~]# systemctl start network
[ 5203.845914 ] igb 000:06:00.1 em4: igb: em4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

/etc/sysconfig/network:

Code:

NETWORKING=yes

/etc/sysconfig/networking-scripts/ifcg-em4:

Code:

DEVICE=em4
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.x.x.x
NETMASK=255.255.255.0
HWADDR=x:x:x:x
NM_CONTROLLED=no

I have checked /etc/udev/rules.d/70-persistent-net.rules, there is no such file.

I bolded a line above for emphasis only. You say they're clones...based on that, and the error message, have you changed the hardware address in the networking scripts? If the MAC address doesn't match, the hardware won't come up, but the device itself may initialize later in the boot process, and 'present', letting you bring it up. Just a guess.

I'd remove the preset network scripts, and bring the box up with no networking, and verify addresses/initialization of the hardware first, and proceed from there.

deathsfriend99 · 02-10-2021, 03:43 PM

The hardware address is correct.
Full disclosure, these aren't exactly clones, they're OpenHPC/Warewulf nodes.
The network devices are created during the Warewulf bootstrap, and the HWADDR is added during that provisioning process.
Although that did get me thinking.
I did a

Code:

ip a

and all the network devices were named eno1, eno2, eno3, eno4, as opposed to the em1, em2, em3, em4 in the configuration.
I thought this might be an issue (although I don't pretend to know why), so I changed the ifcfg to match that and rebooted.

That did not solve the issue.

I did, however, reimage this machine using the warewulf tools, and in a stateless mode, the network works on boot, but when I switch to a stateful mode, it does not.
This might be warewulf specific. I'll post on a warewulf board and see if there are any solutions.

JockVSJock · 02-14-2021, 08:27 PM

You can use the following command to view the network configs, provides a cleaner interface.

Code:

nmtui

You can use different nmcli commands to get some info, maybe this will give better insight:

Code:

nmcli 
nmcli con status

I'm not familiar with Warewulf. Maybe some of the ethtool commands might be helpful in troubleshooting.