random probe order of NICs in kernel
 

The kernel (or probably more specifically, the drivers) will still probe network interfaces in random orders. Likely this is just race conditions of how fast the various hardware responds. Udev is supposed to help with this, but fails when MAC addresses change.

I guess I need to design a better network manager that can deal with things like this. But, before I jump into that, I should search and ask around if anyone has already done such a thing, or if others have network issues they might like to see it help solve for them.

Things that can change and not be the same:

1. Which physical NIC gets which interface name (can be reassigned).

2. Which MAC address a NIC has (because cards can be changed or the whole physical machine can be changed by moving system drives around).

3. Which LAN is plugged into which port (either at the machine or at the switch racks).

What is to be accomplished:

1. Every network that access is needed to be configured.

2. Proper routes established to other networks (routing/path protocols can help here).

DHCP is, in theory, one solution. But it doesn't handle redundancy well. For example, when 2 or more interfaces go to the same LAN, it has failed to configure all the interfaces, even though they are all up. DHCP also has some security risks which require more complex configuration to resolve.

When broadening the view to see outside the box, it seems that statically configured network addresses really should not be configured in relation to a specific interface name or MAC address, but instead, to an observable network. The kernel will already answer on one interface, an ARP query for an IP address that is bound to a different interface. I've even played with this by configuring lame IP addresses on ethernet interfaces and configured the intended IP addresses on the "lo" interface. It even worked. But I don't know all the issues with it since I have not tried to deploy it in any production use.

The scheme that really seams to make the most sense is to just listen on each network interface and see what the traffic suggests. The goal at this is to learn which LAN/subnet each interface has access to (if any). This could be based on a variety of things seen. ARP queries and answers can be one clue. DHCP probing (ask for an IP address and see what comes back) can be another. Other broadcasts on the LAN can also give some info. Once it is determined what subnet an interface has access to, then this host's configuration in relation to that subnet can now be applied to that interface.

The name eth0 can be assigned to a card by MAC address in:
/etc/sysconfig/network-scripts/ifcfg-eth0

You want your config to stay the same even if the hardware changes. This doesn't seem possible. I think for any suggestion you can make, there are obvious things that will break it.

For example, why go to the trouble of allowing any network cable to plug into any NIC?
It isn't hard to make 3 labels: INSIDE, OUTSIDE, DMZ (or whatever) and stick them over the plugs. I've set up several firewalls this way.

As for changing hardware, NICs don't break very often. When they do, its not hard to change the MAC address in one file.

And for moving system drives around. Wow. How often do you do that? I've done brain transplants a couple of times to upgrade hardware, but not very often.

You sound like a real experimenter. Maybe I'm more from IT background where you like your hardware to stay where you put it.

I have been thinking about this for a few days. The unknown hardware is the stinker. DCHP would be pretty easy.

It may be that some resource also on each net would be available to be used to config. One concept may be to seek some file server or such for it's config. Like let it boot to eth0,1,2 for example and then let one by one look at the lan for some resource that would have to be generic and outside of each lan. When it finds the resource it then configs the eth.

But that won't completely help because the MAC address might be different.

I want my config to relate between the perspective of how it understands the network (e.g. what IP address(es) it uses) and the actual network.

It's not a trouble to allow it. It's more trouble to prevent it. But these are not firewalls. These are computer servers running on more than one internal LAN segment.

It's hard to do it from remote if you can't get into the machine at all. This is about "reliability" ... e.g. the machine "must come up and be reachable" short of being broken. And just changing things around is not "broken". A system (e.g. the personality set up on the system drive) needs to be able to "just work" anywhere.

It varies. Maybe 3 to 20 times a month. It depends on what's going on. Most recent one was to swap systems between 2 machines where one at 4 cores and one at 6 cores and the system that was on the 4 core would greatly benefit from 6 cores, wheres the one on the 6 cores would not. Sometimes it even happens offsite.

It's not experimenting. But I guess I will end up developing a smarter network manager.