This is going to be a long email and I have a work around that works, so don't feel the need to read through this (for those of you busy but helpful people) but I would love to figure this out...

I have built a machine that I use as a router (among other things). I built it with Fedora 11 using iptables for firewalling and routing and dnsmasq for dns and dhcp. I primarily built this so that I can put in squid, squidguard, dansguardian, etc (for those who are quick to point out that a linksys router does routing very well and with much less hassle) and, secondly, because I like tinkering with the stuff.

Here is my setup: I have a dual homed machine with one nic pointing to a DSL modem (WAN) and the other to a switch (LAN). I use IP pass through on the modem so that the IP address of the router is the IP address assigned to me via DSL. Because of this, nic card on the modem side (eth0) is assigned a fixed IP address (the modem handles pppoe internally). The gateway for eth0 is the gateway assigned by AT&T.

The second nic card (eth1) is using a fixed IP to support the LAN and the gateway for it is the IP address of eth0.

When I built the box, I first set up eth0 as described above using Network Manager (nm); it connected and I was able to access the internet fine. Next, I brought eth1 up and I was no longer able to access the internet. Once nm has control, there is no way to disable a single nic so the only way to take eth1 out of the mix was to change eth1 so that it is not managed by nm. When I did that, I was able to restart eth0 and all worked. I then activated eth1 and everything worked! Nice!

So I changed the configuration back to have eth1 managed by nm and everything broke again. What!? I figured, no problem, I would get rid of nm all together and manage the network directly. I moved both interfaces off of nm and disabled it. Started up the network and no joy. Turns out the only way to get it to work is to have eth0 managed by nm and eth1 not... very odd.

I did more looking and when I run /sbin/route I can see when the problem occurs. For some reason, when I have both interfaces managed the same way, the default gateway does not get set correctly in the routing tables. Specifically no default gateway gets set unless I handle the interfaces this way.

I think I have done all the obvious stuff: checking ifcfg-eth[0,1] to make sure they are right; network to make sure it has the gateway... all looks correct. My work around is to have eth0 managed by nm and I put /etc/init.d/network restart into my /etc/rc.local file. This seems to allow nm to do whatever it needs around eth0 and then the restart brings down eth0 and brings the whole thing back up... all is good.

If anyone has any insight on what I might do to figure this out I would love to hear it. As I said at the top, my work around is working, but I don't understand why and that is really the question so that I can learn what is going on. Thanks all.