[SOLVED] HELP!! I cant get the static route configured properly on this second NIC!

linuxr00lz2013 · 10-31-2013, 01:54 PM

hello all

I am trying to configure a second network interface on a server of mine and I am running into some problems when it comes to configuring the static route. The first NIC is for the external network and the second NIC will be used for backup.

This is what my /etc/sysconfig/network-scripts/ifcfg-eth0 file looks like:

Code:

DEVICE=eth0
BOOTPROTO=static
BROADCAST=10.146.112.255
HWADDR=00:21:5E:DB:A9:40
IPADDR=10.146.112.42
NETMASK=255.255.255.0
NETWORK=10.146.112.0
ONBOOT=yes

and this is what my ifcfg-eth1 file looks like:

Code:

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
BOOTPROTO=static
HWADDR=00:21:5E:DB:A9:42
IPADDR=10.146.123.6
NETMASK=255.255.255.0
#GATEWAY=10.146.123.1
ONBOOT=yes

I am configuring the eth1 card on this box. I created a route-eth1 file :

Code:

GATEWAY1=10.146.123.1
NETMASK1=255.255.255.0
ADDRESS1=10.146.123.0

I did a service network restart and the interface came up fine but when I checked the routing using route -n:

Code:

# /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.146.112.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.146.123.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.146.112.1    0.0.0.0         UG    0      0        0 eth0

it seems to only show the default gateway of the first NIC, eth0 and it does not show the route of the second NIC, eth1. I noticed that if I uncomment the GATEWAY section of ifcfg-eth1 , the output is as follows:

Code:

]

# /sbin/route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.146.112.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.146.123.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.146.123.1    0.0.0.0         UG    0      0        0 eth1

It seems to have replaced the gateway for eth0 with the gateway for eth1.

Basically I dont know if I am going about this the right direction? eth1 does not have any nameservers assigned to it. I was instructed not to use DNS for eth1.

Am I doing this correctly or did the original route get replaced with the route for eth1?

All help is GREATLY appreciated!

Thanks!

oh I am using RHEL 5.9

Robhogg · 10-31-2013, 03:18 PM

A couple of issues - this says that to reach the 10.146.123.0/24 network, packets should be routed via the 10.146.123.1 gateway - however, eth1 is directly connected to that network (so a route and gateway do not need to be defined):

Quote:

Originally Posted by linuxr00lz2013

Code:

GATEWAY1=10.146.123.1
NETMASK1=255.255.255.0
ADDRESS1=10.146.123.0

You may be able to define a default route in this file by using ADDRESS1=0.0.0.0 and NETMASK1=0.0.0.0. On the other hand, the CEntOS 5 docs give the syntax for default gateways as default 10.146.123.1 dev eth0

The second issue is that the system expects there to be a single default gateway (by default). Not tried configuring more than one before (the secondary interfaces on the servers I've managed are strictly for backup networks or failover), so can't vouch for the instructions in this blog post, but they may help.

Medievalist · 10-31-2013, 05:34 PM

Most people just put one GATEWAY line in either /etc/sysconfig/network or /etc/sysconfig/network-scripts/ifcfg-eth0.

Many years ago - very early linux kernels - you could have as many default routes as you had interfaces, and it worked perfectly. I ran over 30 multihomed hosts, each with four default routes, and loved it. The linux network stack of that time, which I think came from SUCS, was one of the reasons I started using it.

But now if you try that and run tcpdump on the output you will be pretty horrified. The packets go all over the damn place. DO NOT LIKE.

So you can't easily have multiple defaults any more unless you use policy routing, or run some kind of routing daemon like gated. Unfortunately some opportunistic clods bought the gated sources from Merit and closed the formerly free open source program so I would never use that, and zebra/quagga et al seem to be pretty raw and unfinished. Policy routing to the rescue.

Policy routing discussion here-> http://kindlund.wordpress.com/2007/1...utes-in-linux/

You can implement policy routing on Red Hat/Centos that will survive a reboot by modifying the ifcfg-eth?, rule-eth?, and route-eth? files found in /etc/sysconfig/network-scripts. The explanation that follows was written by the late Lou Goddard; he was configuring RHES5 at the time.

THIS IS FOR PROS ONLY

/etc/sysconfig/network-scripts/ifcfg-eth? and /etc/sysconfig/network must not contain any GATEWAY entries.

You must have a rule-eth? file for each ifcfg-eth? file.
Each /etc/sysconfig/network-scripts/rule-eth? file should contain the following line:

from xxx.xxx.xxx.0/24 tab B priority C

* xxx.xxx.xxx = the first three octets of the IP address of this ethernet interface

* B = ethN + 1

* C = arbitrary priority, I simply started at 500 for eth0 and incremented by 100 per interface

So, in my own rule-eth0 you will find:

from 192.168.10.0/24 tab 1 priority 500

And in rule-eth1 you will find:

from 192.168.20.0/24 tab 2 priority 600

And so on......

Next, let's look at the route-eth? files. They contain the following:

default via yyy.yyy.yyy.yyy dev ethN tab B

* yyy.yyy.yyy.yyy = the default route associated with this network interface

* B = ethN + 1

* N = the ethernet interface you're working on.

So, in route-eth0 you will find:

default via 192.168.10.12 dev eth0 tab 1

And so on....

To check your work you may run the following commands and examine the output.

[root@mrwizard ~]# ip route show

192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.3

192.168.20.0/24 dev eth1 proto kernel scope link src 192.168.20.3

192.168.30.0/24 dev eth2 proto kernel scope link src 192.168.30.3

192.168.40.0/24 dev eth3 proto kernel scope link src 192.168.40.3

169.254.0.0/16 dev eth3 scope link

default via 192.168.10.12 dev eth0

[root@mrwizard ~]# ip route list ta 1

default via 192.168.10.12 dev eth0

[root@mrwizard ~]# ip route list ta 2

default via 192.168.20.12 dev eth1

[root@mrwizard ~]# ip route list ta 3

default via 192.168.30.12 dev eth2

[root@mrwizard ~]# ip route list ta 4

default via 192.168.40.12 dev eth3

[root@mrwizard ~]# ip rule list

0: from all lookup 255

500: from 192.168.10.0/24 lookup 1

600: from 192.168.20.0/24 lookup 2

700: from 192.168.30.0/24 lookup 3

800: from 192.168.40.0/24 lookup 4

32766: from all lookup main

32767: from all lookup default

linuxr00lz2013 · 11-05-2013, 01:11 PM

Quote:

Originally Posted by Medievalist

Most people just put one GATEWAY line in either /etc/sysconfig/network or /etc/sysconfig/network-scripts/ifcfg-eth0.

Many years ago - very early linux kernels - you could have as many default routes as you had interfaces, and it worked perfectly. I ran over 30 multihomed hosts, each with four default routes, and loved it. The linux network stack of that time, which I think came from SUCS, was one of the reasons I started using it.

But now if you try that and run tcpdump on the output you will be pretty horrified. The packets go all over the damn place. DO NOT LIKE.

So you can't easily have multiple defaults any more unless you use policy routing, or run some kind of routing daemon like gated. Unfortunately some opportunistic clods bought the gated sources from Merit and closed the formerly free open source program so I would never use that, and zebra/quagga et al seem to be pretty raw and unfinished. Policy routing to the rescue.

Policy routing discussion here-> http://kindlund.wordpress.com/2007/1...utes-in-linux/

You can implement policy routing on Red Hat/Centos that will survive a reboot by modifying the ifcfg-eth?, rule-eth?, and route-eth? files found in /etc/sysconfig/network-scripts. The explanation that follows was written by the late Lou Goddard; he was configuring RHES5 at the time.

THIS IS FOR PROS ONLY

/etc/sysconfig/network-scripts/ifcfg-eth? and /etc/sysconfig/network must not contain any GATEWAY entries.

You must have a rule-eth? file for each ifcfg-eth? file.
Each /etc/sysconfig/network-scripts/rule-eth? file should contain the following line:

from xxx.xxx.xxx.0/24 tab B priority C

* xxx.xxx.xxx = the first three octets of the IP address of this ethernet interface

* B = ethN + 1

* C = arbitrary priority, I simply started at 500 for eth0 and incremented by 100 per interface

So, in my own rule-eth0 you will find:

from 192.168.10.0/24 tab 1 priority 500

And in rule-eth1 you will find:

from 192.168.20.0/24 tab 2 priority 600

And so on......

Next, let's look at the route-eth? files. They contain the following:

default via yyy.yyy.yyy.yyy dev ethN tab B

* yyy.yyy.yyy.yyy = the default route associated with this network interface

* B = ethN + 1

* N = the ethernet interface you're working on.

So, in route-eth0 you will find:

default via 192.168.10.12 dev eth0 tab 1

And so on....

To check your work you may run the following commands and examine the output.

[root@mrwizard ~]# ip route show

192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.3

192.168.20.0/24 dev eth1 proto kernel scope link src 192.168.20.3

192.168.30.0/24 dev eth2 proto kernel scope link src 192.168.30.3

192.168.40.0/24 dev eth3 proto kernel scope link src 192.168.40.3

169.254.0.0/16 dev eth3 scope link

default via 192.168.10.12 dev eth0

[root@mrwizard ~]# ip route list ta 1

default via 192.168.10.12 dev eth0

[root@mrwizard ~]# ip route list ta 2

default via 192.168.20.12 dev eth1

[root@mrwizard ~]# ip route list ta 3

default via 192.168.30.12 dev eth2

[root@mrwizard ~]# ip route list ta 4

default via 192.168.40.12 dev eth3

[root@mrwizard ~]# ip rule list

0: from all lookup 255

500: from 192.168.10.0/24 lookup 1

600: from 192.168.20.0/24 lookup 2

700: from 192.168.30.0/24 lookup 3

800: from 192.168.40.0/24 lookup 4

32766: from all lookup main

32767: from all lookup default

Hello! Thank you for your response! I took your advice and this is what ive done:

this is my /etc/sysconfig/network-scripts/ifcfg-eth0 file:

Quote:

DEVICE=eth0
BOOTPROTO=static
BROADCAST=10.146.112.255
HWADDR=00:21:5E

B:A9:40
IPADDR=10.146.112.42
NETMASK=255.255.255.0
NETWORK=10.146.112.0
ONBOOT=yes

and ifcfg-eth1:

Quote:

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
BOOTPROTO=static
HWADDR=00:21:5E

B:A9:42
IPADDR=10.146.123.6
NETMASK=255.255.255.0
NETWORK=10.146.123.0
#GATEWAY=10.146.123.1
ONBOOT=yes

my /etc/sysconfig/network file:

Quote:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=blahblah
#GATEWAY=10.146.112.1

my rule-eth0 file:

Quote:

]# cat rule-eth0
from 10.146.112.0/24 tab 1 priority 500

rule-eth1:

Quote:

]# cat rule-eth1
from 10.146.123.0/24 tab 2 priority 600

my route-eth0 file:

Quote:

]# cat route-eth0
default via 10.146.112.1 dev eth0 tab 1

my route-eth1 file:

Quote:

# cat route-eth1
default via 10.146.123.1 dev eth1 tab 2

now when I run , ip route show , I get the following output:

Quote:

# /sbin/ip route show
10.146.112.0/24 dev eth0 proto kernel scope link src 10.146.112.42
10.146.123.0/24 dev eth1 proto kernel scope link src 10.146.123.6
169.254.0.0/16 dev eth1 scope link

ip route list ta 1 outputs:

Quote:

# /sbin/ip route list ta 1
default via 10.146.112.1 dev eth0

ip route list ta 2 outputs:

Quote:

# /sbin/ip route list ta 2
default via 10.146.123.1 dev eth1

when I tried to ping within the LAN and it pings file, but when I try to ping outside of the LAN , it says network unreachable. Please keep in mind that the second NIC has no DNS . I was instructed not to use a second DNS for the second NIC. Now I noticed that the last line of the ip route show output:

Quote:

10.146.112.0/24 dev eth0 proto kernel scope link src 10.146.112.42
10.146.123.0/24 dev eth1 proto kernel scope link src 10.146.123.6
169.254.0.0/16 dev eth1 scope link

looks totally different than expected.. the last line says: 169.254.0.0/16 dev eth1 scope link. It is missing the default via XXX.XXX.XXX.XXX dev eth1 (or eth0) line.. What am i doing wrong here and why is it not pinging 10.146.123.6 from outside of its LAN and why is it not able to ping something like www.yahoo.com but it can ping other nodes on its LAN?

I hope my questions are making sense

Thanks!

Medievalist · 11-06-2013, 10:17 AM

Quote:

when I tried to ping within the LAN and it pings fine, but when I try to ping outside of the LAN , it says network unreachable. Please keep in mind that the second NIC has no DNS . I was instructed not to use a second DNS for the second NIC.

Your instructions were correct, DNS doesn't attach to NICs. You should use at least two and preferably three separate DNS servers (see man pages for /etc/resolv.conf and /etc/nsswitch.conf) no matter how many NICs you have. But I don't think you have a DNS problem.

Quote:

Now I noticed that the last line of the ip route show output [...] looks totally different than expected.. the last line says: 169.254.0.0/16 dev eth1 scope link. It is missing the default via XXX.XXX.XXX.XXX dev eth1 (or eth0) line.

The 169.254 block is to support Apple-style zero configuration silliness, which I think Red Hat/Centos implements with the Avahi daemons. If you aren't running Avahi and you're not on an Apple infrastructure you can turn it off by adding "NOZEROCONF=yes" to your /etc/sysconfig/network file.

Quote:

What am i doing wrong here and why is it not pinging 10.146.123.6 from outside of its LAN and why is it not able to ping something like www.yahoo.com but it can ping other nodes on its LAN?

You did everything right. Now when packets come in to your system, they will be answered on the same NIC they came in on, which is pretty important.

So, do you want to use a single network gateway leading out to the Internet, or do you want to do load balancing, failover, or both using multiple gateways?

If there's only one way to get out, just add GATEWAY=xxx.xxx.xxx.xxx to /etc/sysconfig/network and you are done. (That should have been in Lou's instructions, but wasn't - my apologies.) If you want load balancing, or failover or both, it will be pretty complicated (I haven't ever done load balancing in linux BTW) so if you can use one gateway you should.

More info here: http://lartc.org/howto/lartc.rpdb.mu...ks.html#AEN298

linuxr00lz2013 · 11-21-2013, 01:30 PM

Thanks guys this worked!! At least thats what my customer is saying!

dwfa · 04-15-2014, 10:16 AM

Hi,

I followed the instructions on this thread (which was very helpful by the way) to help met setup my linux box with two nic cards. I ran into a bit of an issue with my DNS service that runs on the same box and not sure where the problem lies. I have isolated to the rule-eth0, ifcfg-eth0, and network files. I was hoping someone can help me understand what I am doing wrong. If you need more info please let me know.

If I use the vanilla (CENT OS 6) generated files everything works. DNS works just dandy. I can query my own servers and query the real world.
ifcfg-eth0

Quote:

DEVICE=eth0
HWADDR=<masked out>
NM_CONTROLLED=yes
ONBOOT=yes
IPADDR=192.168.1.1
BOOTPROTO=none
NETMASK=255.255.0.0
TYPE=Ethernet
GATEWAY=192.168.1.254
DNS1=192.168.1.1
IPV6INIT=no
USERCTL=no

network

Quote:

NETWORKING=yes
HOSTNAME=ds9.dwfa.ca

route-eth0 does not exist

Now if I update the three files to put the default gateway rule into network, ifcfg-eth0 and rule-eth0, my DNS stops working i.e. I get no response from my queries.

ifcfg-eth0

Quote:

DEVICE=eth0
HWADDR=<masked out>
NM_CONTROLLED=yes
ONBOOT=yes
IPADDR=192.168.1.1
BOOTPROTO=none
NETMASK=255.255.0.0
TYPE=Ethernet
#GATEWAY=192.168.1.254
DNS1=192.168.1.1
IPV6INIT=no
USERCTL=no

network

Quote:

NETWORKING=yes
HOSTNAME=ds9.dwfa.ca
GATEWAY=192.168.1.254
GATEWAYDEV=eth0 <--- have tried with and without this entry

route-eth0 I created

Quote:

default via 192.168.1.254 dev eth0 onlink
# default via 192.168.1.254 dev eth0 tab 1 <--- have tried each entry on their own

My DNS config looks like this

Quote:

options {
allow-query { 192.168.0.0/16; };
allow-recursion { 192.168.0.0/16; };
listen-on port 53 { 192.168.1.1; };
directory "/etc";
pid-file "/var/run/named.pid";
forwarders {
# the following are two DNS servers for TWC
209.18.47.61;
209.18.47.62;
};
forward first;
recursion yes;
};

logging {
channel logMessages{
file "/var/log/named.log" versions 3 size 2m;
severity info;
print-severity yes;
print-time yes;
print-category yes;
};
category default{
logMessages;
};
};

zone "." {
type hint;
file "/etc/db.cache";
};

zone "<my domain>" in {
type master;
file "/database/<my domain>.hosts";
};

zone "168.192.in-addr.arpa" in {
type master;
file "/database/<my domain>.rev";
};

Medievalist · 04-15-2014, 11:12 AM

Won't be able to help much, I'm afraid, because I'm about to go off-net for a couple weeks.

But whenever I want a machine to query the local host for DNS, I just put this line in /etc/resolv.conf:

Quote:

nameserver 127.0.0.1

and this line in /etc/nsswitch.conf:

Quote:

hosts: files dns

I don't put DNS details into /etc/sysconfig/network-scripts in any case. /etc/sysconfig can only work on Red Hat clones, but the /etc/resolv.conf and /etc/nsswitch.conf method works on all *ix systems including HP-UX, Sun, the BSD family, etc. etc.

Note there can be two or three "nameserver" lines in /etc/resolv.conf, but there can only be one "hosts" line in /etc/nsswitch.conf.

dwfa · 04-15-2014, 11:46 AM

Thanks for the reply and have fun!

The DNS query does not work (stops responding) from any machine on my network. It works with the default config, but when I change the files above it stops responding.

Quote:

Originally Posted by Medievalist

Won't be able to help much, I'm afraid, because I'm about to go off-net for a couple weeks.

But whenever I want a machine to query the local host for DNS, I just put this line in /etc/resolv.conf:

and this line in /etc/nsswitch.conf:

I don't put DNS details into /etc/sysconfig/network-scripts in any case. /etc/sysconfig can only work on Red Hat clones, but the /etc/resolv.conf and /etc/nsswitch.conf method works on all *ix systems including HP-UX, Sun, the BSD family, etc. etc.

Note there can be two or three "nameserver" lines in /etc/resolv.conf, but there can only be one "hosts" line in /etc/nsswitch.conf.

dwfa · 04-30-2014, 05:32 PM

I got the DNS thing working properly. Now, I am still stuck at getting the routes configured correctly. Here is what I have so far.

/etc/sysconfig/network

Code:

NETWORKING=yes
HOSTNAME=ds9<hiding domain name>
GATEWAY=192.168.1.254

/etc/sysconfig/network-scripts/ifcfg-eth0

Code:

DEVICE=eth0
HWADDR=<hiding MAC>
NM_CONTROLLED=no
ONBOOT=yes
IPADDR=192.168.1.1
BOOTPROTO=none
NETMASK=255.255.0.0
TYPE=Ethernet
DNS1=192.168.1.1
IPV6INIT=no
USERCTL=no

/etc/sysconfig/network-scripts/ifcfg-eth1

Code:

DEVICE=eth1
HWADDR=<hiding MAC>
NM_CONTROLLED=no
ONBOOT=yes
IPADDR=192.168.2.10
BOOTPROTO=none
NETMASK=255.255.0.0
TYPE=Ethernet
IPV6INIT=no
USERCTL=no

/etc/sysconfig/network-scripts/rule-eth0

Code:

from 192.168.1.0/24 tab 1 priority 500

/etc/sysconfig/network-scripts/rule-eth1

Code:

from 192.168.2.0/24 tab 2 priority 600

output from ip rule show

Code:

0:      from all lookup local
500:    from 192.168.1.0/24 lookup 1
600:    from 192.168.2.0/24 lookup 2
32766:  from all lookup main
32767:  from all lookup default

I used iftop to monitor traffic on eth1 while I transferred a large file from another machine on the network (used scp to copy 100MB file), I connected to 192.168.2.10. I saw no traffic on eth1; however; I did the same test again to eth1 while monitoring eth0 and the traffic showed up. Any idea what I am missing; I am sure I made a mistake somewhere.

dwfa

Posts: 13
Joined: 2012/02/29 05:04:15
Top

dwfa · 06-09-2014, 01:36 AM

Any ideas?