LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS
User Name
Password
CentOS This forum is for the discussion of CentOS Linux. Note: This forum does not have any official participation.

Notices


Reply
  Search this Thread
Old 09-27-2017, 09:17 AM   #1
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Rep: Reputation: 16
Network configuration gets messed up after boot


Hi,

on a freshly installed centos 7.3 system two network ports were configured for bonding. which worked perfectly until the next reboot.

networkmanager is disabled and masked, so from this side there should be no problem.

however after the reboot the ifcfg files of the bonded ports were overwritten by a stupid process called dracut. the whole configuration is messed up the bond interface and one physical interface get configured with the same ip adress, the /etc/resolve.conf gets overwritten but not with the search suffixes from the ifcfg files.......

building a new initrd during a state where the network configuration is working doesnt help.


does anybody know how to stop this dracut process from messing up the configuration during each reboot of the computer?


since an update to centos 7.4 the same mess happenes on some of our desktop stations the whole ipv6 configuration gets overwritten each reboot.

many thanks
campher
 
Old 09-28-2017, 12:19 AM   #2
jsbjsb001
Senior Member
 
Registered: Mar 2009
Location: Earth? I would say I hope so but I'm not so sure about that... I could just be a figment of your imagination too.
Distribution: Currently OpenMandriva. Previously openSUSE, PCLinuxOS, CentOS, among others over the years.
Posts: 3,440

Rep: Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784
This looks from my reading to be a known problem, from what I'm understanding, dracut writes the initramfs image. You can read more about it here: https://dracut.wiki.kernel.org/index.php/Main_Page

You can try and add the following to your kernel command-line and see if that helps;

Code:
networkstatic=yes
Let us know how it goes...
 
1 members found this post helpful.
Old 09-28-2017, 01:37 PM   #3
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Original Poster
Rep: Reputation: 16
Thank you it worked, at least for the computer with the bonding interface.
the centos 7.4 stations we still have to test.

campher
 
Old 09-29-2017, 08:58 AM   #4
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Original Poster
Rep: Reputation: 16
Hi,

on centos 7.4 this trick doesnt work. the network configuration gets overwritten even with the bootparameter "networkstatic=yes" set.

i am wondering where this stupid dracut process takes its wrong information from. the static ip address set during installation of the system is nowhere in the configuration files of dracut. but it overwrites the configuration of the network with this address. additionally it wipes out every ipv6 configuration.

is there any other way to stop dracut from messing around with the network configuration?

campher
 
Old 09-29-2017, 11:57 AM   #5
jsbjsb001
Senior Member
 
Registered: Mar 2009
Location: Earth? I would say I hope so but I'm not so sure about that... I could just be a figment of your imagination too.
Distribution: Currently OpenMandriva. Previously openSUSE, PCLinuxOS, CentOS, among others over the years.
Posts: 3,440

Rep: Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784
I was hoping that would also work with CentOS 7.4, but anyhow try the following command;

Code:
systemctl mask rhel-import-state.service
Run the above as root, of course.

From what I'm understanding, it appears that this service copy's the network configuration from initramfs. I had a look at /lib/systemd/system/rhel-import-state.service (the service in question) and it appears (along with some other research I done) that if this service is disabled, it may stop your network configuration from being overwritten.

I have no way to test it myself, so let us know if it works for you.

Last edited by jsbjsb001; 09-29-2017 at 12:06 PM. Reason: typo
 
Old 10-20-2017, 09:40 AM   #6
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Original Poster
Rep: Reputation: 16
Code:
systemctl mask rhel-import-state.service
this service seems not to be active on our systems or it has no effect.



after a lot of testing the whole thing seems to be still quite a mess.

the kernel parameter leads to doubled devices after the boot process on centos 7.4. however the ifcfg files don t get overwritten any more but the system is nearly unusable until a "systemctl restart network" command is done.

a not so invasive solution to the problem is described here:
https://www.centos.org/forums/viewto...p?f=50&t=64258
this is working on centos 7.3 for the bonding interfaces and the ipv6 configuration. this survives even an update to centos 7.4. however when this solution is done on a freshly installed centos 7.4 the whole system gets unusable after a restart. each command needs 5 minutes to execute until a "systemctl restart network" command is done.

so after booting the whole network configuration is a complete mess until the "systemctl restart network" command is done.

dracut is still messing up the system.
very early in the boot process, dracut is starting dhclient and configuring the network devices.

(as far as i could find out this is a "feature" to enable the rare case of mounting the root filesystem over the network. however this is enabled per default)

we have dhcp servers on our network but these special computers with the bonding interface should be configured locally. therefore dhclient is not enabled but dracut is starting it anyway.
dracut is configuring the interfaces with dhclient even with all the tricks to stop this stupid behaviour mentioned above.

later in the boot process the bonding interface is configured. but instead of stopping the devices and reconfiguring them they are directly placeed into the bonding with their whole previous configuration intact.

the bonding interface has an ip address and each physical interface has still its own ip adress from the dracut configuration after the boot process is finished.
this leads to a messed up system (not sure why each command needs 5 minutes when the network is in this state)

the only way to rescue the system is to perform a "systemctl restart network" command and even this needs 5 minutes to get executed. afterwards the network is set up properly.


so the question remains is there a way to stop dracut from messing up the network configuration or is there a way to get rid of this stupid process?

campher

Last edited by campher; 10-20-2017 at 09:44 AM.
 
Old 10-23-2017, 06:51 AM   #7
jsbjsb001
Senior Member
 
Registered: Mar 2009
Location: Earth? I would say I hope so but I'm not so sure about that... I could just be a figment of your imagination too.
Distribution: Currently OpenMandriva. Previously openSUSE, PCLinuxOS, CentOS, among others over the years.
Posts: 3,440

Rep: Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784
Hi campher,

Sorry it's still not working for you, I was hoping that we had finally got it...

A couple of thoughts...

1) When you say "the only way to rescue the system is to perform a "systemctl restart network" command and even this needs 5 minutes to get executed. afterwards the network is set up properly.", that sounds like a hint to me, but I'm not sure where you should look to act upon it though?

2) It seems to me that the solution is either dracut's configuration file or possibly a systemd config/option.

There are a couple of interesting settings in dracut's config file that I think are worth trying, but I would suggest you test it on just one machine and verify it works before going to your other machines;

Try changing /etc/dracut.conf as follows;

Code:
# dracut modules to omit
#omit_dracutmodules+=""
This setting looks like it omit's whatever modules that you specify in it's config file. Based on the link you provided, it seems like this may well be the setting your after.

Try changing the above to this:

Code:
# dracut modules to omit
omit_dracutmodules+="ifcfg"
Let us know how it goes...
 
Old 10-23-2017, 11:10 AM   #8
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by jsbjsb001 View Post
Code:
# dracut modules to omit
omit_dracutmodules+="ifcfg"
thats working for centos 7.3 but not 7.4. the weird thing is when i configure this on a centos 7.3 and update to centos 7.4 it still works. when i install a centos 7.4 and configure afterwards the network is messed up the system gets dead slow until the network is set up properly again.

i even tried a
Code:
# dracut modules to omit
omit_dracutmodules+="dhclient"
but this has no effect.
 
Old 10-23-2017, 12:54 PM   #9
jsbjsb001
Senior Member
 
Registered: Mar 2009
Location: Earth? I would say I hope so but I'm not so sure about that... I could just be a figment of your imagination too.
Distribution: Currently OpenMandriva. Previously openSUSE, PCLinuxOS, CentOS, among others over the years.
Posts: 3,440

Rep: Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784
Well, the more I read about dracut, the more I'm getting a little confused myself. I just had a look at something and I think I should have said this before;

Code:
# dracut modules to omit
omit_dracutmodules+="network"
I have also found some interesting links that may help:

https://access.redhat.com/solutions/3017441

This sounds like it *might* be the solution, BUT you will have to login (and probably pay for RHEL itself) to see the solution. So no help there.

https://www.centos.org/forums/viewtopic.php?t=64197

It sounds like this guy might have had either the same problem as yourself or a very similar one. But unfortunately he's not exactly clear on what the answer is.

https://www.kernel.org/pub/linux/uti....html#_network

Now it *seems* that there are some kernel options for dracut. Which may or may not be any help at all, but maybe worth a try.

While the following link is for RHEL 7, most of it should also apply to CentOS 7;

https://access.redhat.com/documentat...working_guide/

Another idea;

I don't know if this would work, but you could always try building your own initramfs image. Get your machine(s) setup with a working config and then build an image, then get it to use the new image.

Last edited by jsbjsb001; 10-23-2017 at 01:34 PM. Reason: corrections/additions
 
Old 10-27-2017, 10:16 AM   #10
campher
Member
 
Registered: Jul 2006
Distribution: slackware
Posts: 57

Original Poster
Rep: Reputation: 16
Hi jsbjsb001

thank you for your help.

i found a solution
the file to prevent dracut from messing up the network /etc/dracut.conf.d/ifcfg.conf must look like this
Code:
omit_dracutmodules+="ifcfg network"
the "network" part was missing in my previous attempts.

afterwards a
Code:
dracut -fv
to build the init ramdisk and check if the network is really left alone by dracut
during the next reboot dracut does not start the dhclient anymore and the bonding ist set up properly.

campher

Last edited by campher; 10-27-2017 at 10:18 AM.
 
1 members found this post helpful.
Old 10-27-2017, 03:52 PM   #11
jsbjsb001
Senior Member
 
Registered: Mar 2009
Location: Earth? I would say I hope so but I'm not so sure about that... I could just be a figment of your imagination too.
Distribution: Currently OpenMandriva. Previously openSUSE, PCLinuxOS, CentOS, among others over the years.
Posts: 3,440

Rep: Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784Reputation: 1784
Well done campher!

Thanks for the update and the solution.

I'm glad you finally stopped it overwriting your network config.

I thought it was the omit_dracutmodules+="something" setting/option to stop dracut missing up your network config, I said in post #7 that you were after. I just give ya the wrong dracut config file.

Also, thanks for confirming that you need to build the init ramdisk again.

I'm sure this thread will help others.

But given we now know what the solution is and you have therefore solved it; could you maybe mark this thread as "solved" so future readers know there is a solution in this thread?

Thanks again!
 
Old 11-13-2019, 06:28 PM   #12
hunter86_bg
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Rep: Reputation: Disabled
Due to extra NICs (9 ports in total with built-in), dracut was taking more than 5 min to fail getting DHCP for each port .
Removing the 2 modules above + clevis (dracut complains as it relies on network but we omit it) has speeded up powering up the system with more than 5 min.

Code:
omit_dracutmodules+="clevis ifcfg network"
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Boot loader messed up, triple boot setup mase2hot Linux - Laptop and Netbook 23 10-15-2015 07:17 PM
Messed UP Boot Order on dual-boot, don't know GRUB2, need help -- another's PC hilyard Linux - Newbie 4 06-25-2011 12:57 AM
Boot Problem in Grub (Chainloader failing? /boot messed up?) gjemmott Linux - Desktop 2 03-15-2010 10:26 AM
boot/root environment messed up, cannot boot server! Garball Linux - Server 6 10-02-2009 12:46 AM
messed up the configuration, booting to grub parsek77 SUSE / openSUSE 1 07-13-2005 02:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS

All times are GMT -5. The time now is 03:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration