Being unsatisfied by existing network configuration methods for certain common distributions, I've decided I need to write something new to replace them and make network configuration more reliable and straightforward. I'm hoping this might be as simple as a replacement for an init startup script. But it might be beyond what a simple shell script could do.

My idea is to have a script or program that parses a network configuration file with a different arrangement of directives. I believe this structuring is closer to how real networks are organized. These directives would include:
What the subnet directives do is define what the various subnet sizes available are. it will be an error to overlap subnets. The ip directive just defines what IP addresses will be used. These IP addresses will (obviously) be associated with the subnets that contain them. The ifname directive indicates what subnet a named interface is attached to. That interface will be given the IP addresses in the subnet. The hwaddr directive is like ifname, except that it applies to whatever interface has that network hardware address (MAC address for ethernet). The busaddr directive is like ifname, except that it applies to whatever interface has that bus address, as seen in /sys/devices.

This is the simple form. I would also add some ways to fine tune exceptions to this. But in most cases the simple form should do well.

Example:
Any thoughts on this? Anything you think network configuration needs to have?

Among my needs includes the ability to be both interface name and MAC address independent. So that's why I am including the bus address capability.

Just use this software of mine: http://code.google.com/p/badvpn/wiki/NCD
Works on all Linux distros and it much more powerful than anything else in existence, especially compared to a run-once script. A notable feature is proper handling of hotplugging.

NCD is interface name independent as long as you keep your names in a variable, like the examples on the wiki do. However you shouldn't need to change the names in the configuration, because you can tell udev how to assign names to hardware instead.

Udev will assign persistent interface names based on the hardware MAC address; see this. This means that after a network card is first detected, it will be assigned a new name, and this name will be used in the future for this card, no matter where it's plugged in.

If you for some reason want your network config to be done by bus address instead of by MAC address, you can override udev's persistent naming rules with your own. For example, create a file /etc/udev/rules.d/71-my-net.rules (the number overrides 70-persistent-net.rules):
This will make sure the card at PCI bus address 06:00.0 (see lspci) will be named eth33. Don't forget to add the leading 0000:.

EDIT: your rules may conflict with udev's persistent assigning - you have to prevent udev giving other interfaces those names you assign per bus ID. A quick hack is to add placeholder entries in /etc/udev/rules.d/70-persistent-net.rules which reference reserved names, but are never actually matched:

This will reserve the name eth33. Just the NAME part is important, and ATTR{address} which makes sure the rules is never matched. Note that while the rule is never matched by udev itself, its persistent rules generator parses it, sees the name eth33, making sure it won't give it to any new interfaces.

You also have to manually remove other entries in /etc/udev/rules.d/70-persistent-net.rules which assign reserved names (eth33).

Update: I've just added a "bus" variable to NCD's net.watch_interfaces(). This allows for easy per-bus-location configuration of interface, and doesn't require dealing with udev. See this example.

Which is exactly what I do NOT want.

Basically it is what I want, but I don't want to have to code specific bus addresses. I want it to be "bus order consistent". If udev were to do that persistence thing it does with MAC addresses, but instead do it with BUS addresses, then that can help. But even that's not perfect.

What I have done most recently is a script that scans /sys/devices for network interfaces and constructs a new /etc/udev/rules.d/70-persistent-net.rules based on that. I run the following script before starting udev:
I'd rather not be hacking little bits at a time, like this. But for other reasons I find common network configuration methods, at the level where IP addresses are bound to interfaces, to be lacking. My newest idea I started this thread about is a "do over" for such configuration. This idea addresses multiple concerns. But for the interface name change issues, it addresses it be allowing the reference to which interface to be expressed in a variety of ways:

1. By interface name (the usual way)
2. By MAC address
3. By bus address

I'm also looking at doing a 4th method which will be to use the subnet itself as the interface selector. What this would involve is a program that sniffs the traffic on the interfaces to detect subnets in some way. In promiscuous mode, it could detect the subnets by a few ways. A subnet configuration could have default routers and other hosts listed, and it would send ARP queries to see if there are responses. Or it could send a DHCP request and subsequently turn it down. This is all just to detect what subnets each interface has. It would allow more than one interface reaching a subnet (with ultimately the same or different IPs), and more than one subnet (and hence IP) per interface.

And all this can be configured in one file, or an ordered concatenation of files from a subdirectory.

It is this newer means of configuring the IP addresses that is my interest as a solution to both the udev issues (although mostly solved ... see the script above) and IP addressing issues (with my intention to gracefully allow multi-multi associations between subnets and interfaces).

You might also want to make it pattern match.

Udev does MAC addresses by default. Of course you can code up your own persitent-naming scheme. My guess it that it's just a matter of changing /lib/udev/rules.d/75-persistent-net-generator.rules and maybe (probably not) /lib/udev/write_net_rules.

I do not understand why you want such a naming scheme. Why does the default MAC-address based scheme not work for you? The only reason I can think of is when you want to replace an existing card with another one - but then how hard is it to update the interface name in your config, or the persistent naming cache? What problem are you actually trying to solve here?

This is not really very deterministic from a practical view. Adding a new device may shift remaining devices and their names will suddenly change. In fact, determinism is what MAC-based naming provides - the interface name will not ever change. Anything else is less deterministic.

Right, I guess I'm going to add a `mac` variable to NCD's `net.watch_interfaces()`.

NCD already has net.ipv4.dhcp() which can can be used for the purpose of identifying the network. Actually, it's more reliable to identify a network based on the MAC address of the DHCP server than the IP address/subnet. See this question where I provide an example. (However, it is currently not possible to release the DHCP lease - I'll implement this sometime)

I don't think that the ARP probing is a good idea. Nowadays everything runs DHCP and you can use that to obtain the network config (and to identify it, like I said above). However, NCD is very modular, and you can implement an ARP probing statement if you like; see my DHCP client code for how to to link-layer traffic.

That's what NCD does - most of your network config is the NCD script, with some exceptions like wpa_supplicant conf files.

NCD already has regex_match() which can be used with the `bus` variable of `net.watch_interfaces()`. Since I don't know exactly how you meant to use matching, I can't give you an example.

Can you please give me some feedback about NCD? Is there some particular feature it's missing? I'm not fond of the idea of someone making a whole new network config system just to solve one particular problem. Also, it doesn't seem like your proposed system can do more than wired interfaces. It is my belief that any such simple declarative system for network config will have extreme extensibility and expressivity issues. NCD's power comes from its imperative nature, but with just the right amount of declarative style for the problem domain, in particular the reverse-execution thing.

Maybe it needs to be the latter. I want consistent naming, but not necessarily fixed naming. The difference here is subtle, and in most cases won't be an issue. But where it comes to be an issue, the fixed naming scheme based on a MAC address is the bigger issue.

1. The NIC card is changed, as you mention.

2. The whole motherboard or more is changed. More specifically, hard drives are moved from one machine in a rack to another. Perhaps this is because the machine they move from failed.

3. The system lives on a portable flash drive device, such as a USB memory stick.

When the change happens at a remote location, where people executing these changes are less skill or less involved, making the needed network changes is not easy, if even possible. Something smarter in software can and should take over this role.

This can even involve the USB memory stick, which might be the tool used to allow a remote admin to access the machine where no admins of sufficient skill to reconfigure the network are present on site.

I'm wondering at this point how PXE can affect this. But that will need to wait until I get my feet wet with PXE.

I understand the issue, and it isn't all that easy to solve (but see my method #4 approach). But the MAC really isn't deterministic at all.

If I had designed ethernet today, I'd simply have each NIC card generate a random 128-bit UUID on the fly each time it resets, and use that like the MAC. The only reason I wouldn't do that with today's ethernet is the MACs are only 48 bit. My point is, given enough bits, we really don't need to be assigning fixed addresses to specific hardware. That would cut some costs, too.

DHCP servers can change MAC address, too. Now imagine all the configurations that need to be changed if the DHCP server's MAC address changes.

Even where DHCP is running, it can be limited, such as only one subnet where a network has more than one.

How would the network configuration I gave as an example in post #1 be coded for NCD?

One concept I'm using in my configuration model is that an IP address is NOT considered associated with an interface, but rather, with a subnet. The association is strict in that it is only associated with a subnet that it is IN. There can be more than one IP address on this host in a subnet. It is the subnet itself that is associated with the interface. Once that association is established, all of this machine's IP addresses are associated with the interface ... or with all interfaces associated with that subnet. Organizing this way allows for the list of IPs, and the association of subnets, to be configured separately where desired (e.g. at least some of the IPs come from a web server).

Maybe it doesn't make any sense to match a bus address with a regexp. Or maybe it does for someone else. I don't know. If my code has regexp capability, I'd let it be used even when I don't anticipate a need, as long as I don't see an issue that breaks things.

Is NCD not a whole new network config system?

If course there can be potential issues with any system, declarative or imperative. That's one reason to ask for feedback. And yes, the idea I have is perhaps overly simplified. But I would want to have the ability, for a network that CAN be expressed in a simple way, to actually express it in such a simple way. I would not want the ability to do a very complex configuration to impose on simpler networks to express things any less simple than they need.

My initial reading of NCD gives me that feeling that it is, at least in some major part, still based on the concept of "this IP belongs on this interface". I'd rather more have a case of "this IP (these IPs) are what I need to use, now please sort out for me which interfaces they need to go on". If the subnets for each are known, that sure simplifies it, as which subnet a given IP is in is certainly clear. Then you only need to associate a subnet and an interface (or establish all the many-to-many associations that really exist).

In some cases, other programs may need to determine which IPs are used. The ability to express that needs to be simple from the other program's perspective. For NCD, it might be better for the other programs to store their IPs elsewhere, and have a script generate the NCD config from that.

But NCD seems overly mechanical (in the imperative sense) to me. The configuration expresses "what to do" rather than "how it is to be". There's lots of logic in there. I'd rather avoid the logic on less complex networks. I'd rather express what the network is like, even including detection mechanisms like the ARP/DHCP tests (or others), and let the network config daemon figure out how to make it work, and achieve the desired state.

I've coded up an example for you, see attachment. It's simplified a little for clarity (no IPv6, only one subnet per bus or interface name, IPs are declared within their subnets), but it could be extended with little effort. Also, the actual configuration commands are commented out, for ease of testing.
NOTE: you need the latest NCD from the SVN repo for this program.

I should note that this NCD program I wrote is fully hotplug-aware. For instance, you can pull an USB NIC in and out as much as you like, or pull the network cable in and out, and the interface will be de/re-configured automatically. Actually, this ability is a key feature that is implied when using NCD for network config, provided you wrote the NCD program correctly.

It sure is - but it aims to solve a whole bunch of problems, not just a single problem (like your subnet associations).

I think generating an NCD program from a simple declarative configuration is a very good way to go. The program I attached demonstrates how straightforward the translation can be. And NCD does not impose any interface->IP address association. It's just that all the example NCD programs you've seen are based on this concept, because for the majority of cases this is a good model.

The whole idea of NCD is to be a programmable network config deamon. You are correct saying that it's mechanical. But really, it has to be. I think of the NCD's programming language as "assembly language for network configuration". It is my opinion that to implement high-level declarative network configuration, you shouldn't try to jump paradigms and go right down to its imperative implementation (like a C daemon or shell script), but instead take a smaller step by translating it to a low-level network-configutation language (e.g. NCD).

Attached Files

skaparen_simple.txt (6.7 KB, 16 views)

So what each of us is doing is really at a different level, and could even work together. My starting idea was to replace /etc/rc.d/rc.inet1 and /etc/rc.d/rc.inet1.conf in Slackware, to make it easier for multiple IPs per interface, and do the subnet arrangements. Instead of doing that, I could use my scheme to generate or add to the NCD programmable network config.

I suspect lots of people won't want to create whole new NCD configs for what are otherwise simple (even if multiple interface and multiple IP) networks. It's not as bad as coding XML, but for the simple networks, I think my syntax idea is definitely simpler. But if someone has something complicated, like wanting to divide up IP address assignments among interfaces that otherwise reach the same subnet, more detail is needed in the config. And even that isn't very complex at all. NCD looks like it would handle all kinds of complicated things.

I like the comparison with NetworkManager your web description makes. Among my beefs with NM was that it is essentially useless for servers. And that doesn't count the confusion it has with trying to do 2 subnets on one interface. NCD looks like the replacement. If there is a GUI to add on to NCD, then it NCD could replace NM even for those all-GUI environments used by people that want a GUI tool for everything, and still offer a programmable back-end as an alternative (don't try to do everything in a GUI).

It would still be nice to have NCD be able to (at least allow the configuration programming to) deal with things like some newbie admin in a data center transposing the CAT5 connections. Having defined means (not assumed by NCD ... must be configured as policy by the admin) to identify networks that interfaces are connected to ... and the changes that happen (I count changing the NIC port connections as equivalent to hotplugging), would still be valuable. The "network detection policy" programming could allow the use of monitoring for various traffic types (DHCP, ARP, other broadcasts), or probing in certain ways (send DHCP request, send ARP request, send DNS query, and perhaps many other things) to figure out the network topology on the fly, but initially at startup, whenever events are detected (filterable), and on a timed schedule.

I still believe ARP is a valid way to detect possible subnets. Many networks don't run DHCP or don't want to depend on it. ARP can be used by intentionally looking for a specific configured IP address just as if needing to send IP traffic there, or or just monitor for other hosts doing ARP queries, and the answers they get. One could also specifically query an IP address seen by another machine doing an ARP query and not getting an answer. Use NDP for IPv6. And this all needs to be policy configured, and not the default, since there are potential security implications like DoS.

Yes, there's one part where NCD is currently lacking: run-time control and configuration. For the moment, all input to the NCD daemon is the NCD program as it was when the daemon started up. Once it's running, it can mostly only react to network-specific events (like interface status); to change something you have to modify the NCD program and restart the daemon, and with that all the network. It would be beneficial if the daemon could be directly controlled at runtime. For example, in the simple "interface name <--> IP configuration" model, to change the name of the interface that is supposed to have a particular IP configuration. A program would contact the daemon and request a change in configuration, and the daemon would react as the NCD program specifies. This control thing would not be any special construct - it would be implemented as regular NCD statements.

However, control itself is not enough. There should also be a simple way to permanently store such run-time configuration on disk, to load it into the daemon, and retrieve the current config from the daemon.

Having said that, here's an idea for a particular way to implement run-time configuration.

• The NCD daemon has a global attribute, the configuration.
• The configuration is an NCD value - the same thing as statement arguments are. However, for usability's sake, the meaning value would be extended to include a map type. Formally, the new definition of a value would be:
  
  • A string is a value.
  • An ordered list of values is a value.
  • A map, that is, an unordered set of (value1, value2) pairs, with unique value1, is a value.
• On startup, the daemon optionally reads the initial configuration from a file. This file is the same syntax as NCD statement arguments use (see below).
• At runtime, the daemon listens on a Unix socket, through which it exposes its configuration. It allows a client program to download the current configuration (e.g. to a file)
  and to upload a new version (e.g. from a file).
• NCD language statements are implemented to work with the daemon's current configuration. Here's what I think would be a starting point:
  
  • conf.read(string path) would wait for the given configuration attribute to appear, and expose its value (as the "empty string variable", like var() and list()).
    The given string is a path to the value in some syntax; see below for example. The statement would go up when the value becomes available, go down when it disappears, and toggle itself down-up when it changes.
  • conf.foreach(string path, string template_name, string args) would manage a set of processes, with one process for each element of a list or map. It would start and stop processes dynamically
    as the list/map changes. The element of the list or map would be exposed to the template processes as special variables, for example as _listelem (for lists), or _mapkey and _mapvalue (for maps).
  • When a new configuration is uploaded, it would be compared to the old one, such that only the actual changes trigger actions by the above two modules.

A very simple example of a configuration follows. (it is not what you want, just demonstrates the form of the config).
Here, list syntax is {value1, ... valueN} as NCD already implements, and map syntax is [key1 => value1, ..., keyN => valueN].

Also syntax is needed for identifying a value within the configuration (for use in e.g. conf.read(string path)). A filesystem-like syntax is best IMO. For example:

/interfaces/eth0/addrs (refers the the addresses list)
/interfaces/eth0/addrs/0 (refers to the first address, probably that will never be used)

Using this scheme, the "easy configuration for a dummy admin" would simply mean changing the configuration file and invoking the config client to upload the new configuration. The scheme also allows for GUI-based configuration, where a GUI would act as the config client, and it may or may not use the config file syntax used by the command line config client.

Feel free to suggest improvements on this or an alternative.

I'm guessing the "policy" thing would best map to code in the NCD program and to the configuration state I mentioned above (making the policy easier to configure).

I agree here, ARP checking would be a good thing to have. If you don't have any objections, I'll go ahead and try to code an NCD module to do it. Probably something like net.arp.probe(string device, string address) which would be in up state when it thinks that address exists locally on device, and in down state when it thinks that it does not. I suggest that at the beginning it would send some ARP requests with a small interval. If it got a reply, it would get up. If it does not after some tries, it remains down and extends the probe interval (to avoid congestion). Similarly, when it does get a reply and gets into up state, it would decrease the probe interval, and check for unanswered requests. After some unanswered requests, it would temporarily shrink the interval as a last attempt to prove the host exists, and go to down if that fails.

Unfortunately I don't have any IPv6 network and I've never played with it, so I'm not sure if I'll be able to do that.

Also, I think it would be useful to have a config operation that does not change the configuration, but resets a part of it. For example, resetting /interfaces/eth0 would be the same as uploading a new config without eth0, them immediately uploading the original config (atomically). The effect, in this case, would be to restart the eth0 interface entirely.

You can have NCD inform those programs of such parameters (e.g. IPs). Many things can be done using only run() or runonce(). You can also have NCD manage those programs (start them, watch over them and restart if needed); to do that, however, you currently need to implement a custom NCD statement to deal with that program. Existing statements that do this are net.backend.wpa_supplicant() and net.backend.badvpn(). In the future I may add a statement to manage simple daemons, maybe called spawn_process() or daemon().

An advantage of this approach is that the the parameters in question can be dynamic, e.g. an IP address obtained with DHCP.

NCD now has net.ipv4.arp_probe()

In what way does that change the state? The use I'd have for ARP probe would be to add a subnet to the interface the ARP probe succeeded on. I'd want to send this probe on all, or maybe a list of, interfaces, to see where the subnet is. And the subnet may be on more than one interface. Upon success, if the interface is down, bring it up. Then add the IPs that are appropriate. And usually I would associate IPs with a subnet. Also, once the subnet is activated, new IP requests (for example a new vhost added to a web server) should know which interfaces (based on this subnet being active for them, now) to add them to. And those IPs should also be added to the subnet's list of IPs in case the network changes and interfaces go down, come up, or that newbie admin comes around and swaps the cables again (should trigger a lot of events to figure out how to adjust the configuration to work with that).

Sound complicated?

Is there something stopping you from doing that? For a given interface, start a net.ipv4.arp_probe() for each subnet that you expect could appear on that interface. Here's an example. It's hardcoded to a single interface and a static set of subnets each with its single IP, for clarity; see the NCD program I made for you to see how it can be generalized, and the foreach thing.

Sure, just do this for every interface you want to.

A run() after assigning the IP should be able to help with that; or, if the interface to your application is more complex; a new NCD module may have to be implemented.

What list? They'll be just assigned. If the cables are swapped, NCD will detect that and reassign the IPs appropriately. This expected behaviour is implied by the reverse-execution of statements when some statement goes do down state. The NCD wiki explains how that works, http://code.google.com/p/badvpn/wiki...l_of_execution .

Not so much, and the requirements do make sense. I don't think you're fully aware how much NCD can really do. I've added some more docs to the NCD wiki page, you might want to take a look.