[SOLVED] 2 or more ethernet ports - how to make the alternate work
Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
2 or more ethernet ports - how to make the alternate work
I have 2 (some future machines will have 4) ethernet ports. I want to have them configured such that if any gets unplugged, as long as at least one of them is plugged in, it can reach the network (even if via a different IP address), and it can be reached (at least when trying a working IP address).
I tried this for /etc/network/interfaces:
Code:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 172.30.16.3
netmask 255.255.0.0
network 172.30.0.0
broadcast 172.30.255.255
auto eth1
iface eth1 inet static
address 172.30.20.3
netmask 255.255.0.0
metric 33
But that doesn't work. If eth1 gets unplugged, it can no longer reach the net or LAN ... except via IPv6 link-local addresses. Here is the output from "route -n" which looks wrong to me:
Code:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 172.30.0.2 0.0.0.0 UG 1 0 0 eth1
At the time of that "route -n" command, eth1 is unplugged. How can I make the routes "go away" for that situation, and "come back" when it gets plugged back in?
What I believe SHOULD happen is:
1. There should be a default route for at least each of the working interfaces, if not all of them.
2. The metric for the interface not plugged in should be automatically set/faked at its highest possible value to effectively disable it, and restored to its configured value when that interface is plugged back in and working again.
If I unplug eth0 and plug the cable into eth1, then things start working.
Another factor in this: SOMETIMES, after a reboot, the situation is "reversed" where default route specifies eth0 instead of eth1. In that case, it works when eth0 is plugged in and eth1 is not.
Another symptom. When eth1 is unplugged, it takes 60 seconds for the system to shutdown to the point where pings (to the working IPv6 link-local addresses) stop. If both ethernets are plugged in, it only takes about 3 seconds. It appears the init scripts are stalled doing a "stop" of apache. Maybe apache is doing some DNS lookup that won't work ... during a shutdown?
Here's some more info. Note that eth1 does not have the RUNNING state and does not have an autoconfigured IPv6 Scope:link address, since it is the one without a cable plugged in. Also note how the ARP table only has entries for eth1, not eth0, and they are all incomplete (those addresses being router gateway and DNS servers in the LAN).
You know that this is easier with a BSD system. Each card is assigned according to the driver. If you have an amdtek, an intel, and a realtek the cards would be dc0, fxp0, and rl0 respectively.
You know that this is easier with a BSD system. Each card is assigned according to the driver. If you have an amdtek, an intel, and a realtek the cards would be dc0, fxp0, and rl0 respectively.
But would the routes be set to use all of the card ports that have working connections, without trying to ARP on an interface with nothing plugged in?
Long ago I did something like this in Linux and it worked fine. But that was Slackware and now I'm doing it with Ubuntu. Maybe I should try Debian, Fedora, or Gentoo, or go back to Slackware?
Try changing "auto" to "allow-hotplug", if that doesn't work use ifplugd. You could use the same IP and Ethernet addresses on both interfaces.
I don't see any man page for ifplugd, and I don't want to install the package, yet (due to the tendency to start things just because they get installed, and I don't want to do that until after I know what exactly it does). Do you know if it can simply be configured to run any script (or must it use ifup/ifdown)? I have IP aliases and they are NOT configured for ifup to use (because ifup does it wrong). If I can just get ONLY the route entries changed when a cable is (un)plugged, that would do. I was thinking of having a cron script check interfaces and reload routes based on what was RUNNING and what was not. But I'd rather this be done instantly (it should just change the route metrics to highest value when the interface is unplugged and change it back when plugged in).
I can look into using the same IP on each interface. I did that for something else years ago, and it caused a lot of problems. But, this was back when this routing was not broken (e.g. back then, if I unplugged a cable, that interface was not used, and any communication that could work over another interface, did work over another interface). I know the kernel has changed the network logic many times over the years. Maybe shared IPs is fixed, now.
I wish they would fix ifup to handle IP aliases correctly (e.g. NOT treat them as individual interfaces ... which causes it to restart many daemons X times when there are X IP alias addresses ... taking a few seconds each, taking many minutes to bring things up when X is large, and causing some daemons to just shutdown and not come up at all).
I wish they would fix link local in IPv4 to operate like it does in IPv6 (e.g. it won't be used a source address unless the destination is also a link local).
Quote:
Originally Posted by TimothyEBaldwin
But is just checking for a disconnected cable sufficient?
That would be a good start. Going further to check if traffic can actually flow would be harder, since the IP layer would not make a good test when more than one interface is present. If IP can't get through, it's unclear why. What is needed is a way to literally send ethernet frames to a specific interface and get something back from the switch itself. Maybe I can do that through IPv6 link local.
Just to summarize what I THINK the network stack and drivers SHOULD do ... when an interface enters an error or not-operating condition, ALL routes pointing to it should be FLAGGED to operate as if their metric was infinite (or at least the highest possible metric), without actually removing the routes or losing their administratively configured metric (or other data ... e.g. do not remove the route) ... or ... the routing logic should test the device when a route is found, and if the device is not-operational, continue the route search as if it was not a match. If I unplug a cable, there is no reason the ARP table should have any "(incomplete)" entry naming that device (at least not entered after unplugging).
That would be a good start. Going further to check if traffic can actually flow would be harder, since the IP layer would not make a good test when more than one interface is present. If IP can't get through, it's unclear why. What is needed is a way to literally send ethernet frames to a specific interface and get something back from the switch itself. Maybe I can do that through IPv6 link local.
Actually if the switch supports Spanning Tree Protocol, that is easy; just use the bridging. Install bridge-utils and read the bridge-utils-interfaces man page, you should probably increase the bridge priority value to ensure your server doesn't become the root bridge.
Did you mean "man ifconfig"? If so, maybe you can say what part of this command you think would help here.
I can look into the spanning tree. The switch surely supports it (HP ProCurve). But I don't understand how that behaviour would fix this.
I've been trying things out with different configurations with the same IP configured on both interfaces ... with various problems, and I've found that the whole proxy ARP implementation may be FUBAR. It's doing proxy ARP even when turned off. And ... even when eth1 is unplugged which means ARP requests MUST come in on eth0, it sometimes answers them with the MAC for eth1 !!! And I can't turn proxy ARP off ... it was already off.
The only way that makes things work for sure is to just NOT configure eth1 up at all (plugged in or not makes no difference). If eth1 is not configured, there's no point in even having this.
Just to update ... this problem seems to be far more complex than previously thought. It seems that route settings and ARP settings are thoroughly entangled, and not in a good way. And this seems to all be because even when a physical interface is not in a RUNNING state, which the kernel knows about, it still behaves indefinitely as if it were.
There may even be 2 distinct problems, each masking the other from being clear.
It might be fine if the kernel holds off for a minute or so when the interfaces transistions from RUNNING to not. But for HOURS is just wrong. If the interface is not working, the kernel should not be trying to pretend that it is. It should NOT be routing EXCLUSIVELY to that interface.
I think maybe what I need is HSRP or VRRP for Linux.
Because when you unplug or plug interfaces you should do: ifup and ifdown.
Look, you can try to find "man ifcfg" in internet, because, as it was said before here, there is:
Code:
STARTMODE {manual*|auto|hotplug|ifplugd|nfsroot|off}
Choose when the interface should be set up.
manual Interface will be set up if ifup is called manually (without option boot or hot-
plug)
auto Interface will be set up as soon as it is available (and service network was
started). This either happens at boot time when network is starting or via hot-
plug when a interface is added to the system (by adding a device or loading a
driver). To be backward compliant onboot, on and boot are aliases for auto.
hotplug
This mode is nearly the same as auto. The difference between auto and hotplug is
that the latter does not make rcnetwork fail if the interface cannot be brought
up.
ifplugd
The interface will be controlled from ifplugd. At initial ifup only ifplugd will
be started for this interface. Then if ifplugd detects a link if calls ifup again
which finally sets the interface up. See also variable IFPLUGD_PRIORITY below.
nfsroot
Nearly like auto, but interfaces with this startmode will never be shut down via
rcnetwork stop. ifdown <interface> still works. Use this when you use a root
filesystem via network.
off Will never be activated.
Or you can write a small script which will parse output of ethtool <ethN> and check if carrier appears, and if it appears script turn interface UP and DOWN later.
It looks like either hotplug (ifplugd) or port bonding (if my switch can understand it) is the way to go. But I'm not convinced that will fix it all. I still don't know why the kernel was filtering out packets it actually received on eth1, while allowing them on eth0 (both had the same IP address), and prevented the NSD process from getting them.
This stuff worked right the last time I used dual ethernet ports. But that was many years ago and it was on Slackware, not Ubuntu.
So, what I think is needed is a document that simply describes exactly what steps are needed (and lists what extra packages are needed for the major distributions) to make 2 or more ethernet ports work correctly. The searches I have done on this have turned up how to do bonding (not what I wanted to do), how to bridge between them (not useful to me), and many unanswered or inadequately answered questions. In my case this situation involves Ubuntu. But I didn't even find one for Fedora (I could put new servers on Fedora and switch the other, later, if I needed to).
It looks like "man ifcfg" is Fedora/Centos specific. I did a search for it under Ubuntu packages and there is no package that mentions it in any way. Ubuntu has "ifconfig" and "ip" commands, which are also on Debian and Slackware. I don't have Gentoo handy right now to check it.
Yes, script "ifup", which is read configuration file ifcfg <interface> is definitely different on linuxes.
But idea is the same. Try for example, using command ifconfig + ifup to do next thing:
1. Give to an interface IP and activate it.
2. Deactivate it.
3. Then, give THE SAME IP to different port, then activate it (ifup <ethN>)
In Ubuntu it is the /etc/network/interfaces file. This is where I configured both eth0 and eth1. I have tried leaving them down by commenting out the "auto" directive, then manually bring up the interface with "ifup eth1". Other than that, I don't know which step you were suggesting I look at this for. I've tried many things with the various configurations. The routes always show which one was last configured, when they should show both interfaces.
You should not just comment out the "auto", you need to put "manual" there. It prevents script "network" from doing its job.
Then you can configure all interfaces (for example) with the same IP. BUT !!!, bring UP only one, then after putting it DOWN you can UP another interface, but not two at the same time (because they have same IP). It will work like "ifplugd".
Another question, you said:" The routes always show which one was last configured, when they should show both interfaces". You mean "routes" are which you connect ethernet to?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
