I have a RHEL 5 server with dual NICs. The first NIC is setup for my data network (172.20.x.x). The second NIC is setup for my SAN traffic. (172.23.1.x) Here is my route export:

Destination Gateway Genmask Flags Metric Ref Use Iface
172.23.1.0 * 255.255.255.0 U 0 0 0 eth5
169.254.0.0 * 255.255.0.0 U 0 0 0 eth5
172.20.0.0 * 255.255.0.0 U 0 0 0 eth4
default 172.23.1.254 0.0.0.0 UG 0 0 0 eth5

I need the default route to go to 172.20.254.254. I can do the following:
route del default gw 172.23.1.254
route add default gw 172.20.254.254

This works fine until I do a reboot. How do I get these settings to stay after a reboot?

Mike

/etc/sysconfig/network-scripts/ifcfg-eth5

add/change:
GATEWAY=172.20.254.254

Ryan Ivey
Linux Administrator
Initial Source Solutions, LLC
www.initialss.com

Thanks for the quick response. The only problem I have with your suggestion is I am trying to keep the traffic separate. I need traffic bound for 172.23.1.x going out eth5 and I need all other traffic going out eth4.

So then,
edit /etc/sysconfig/network-scripts/ifcfg-eth4 - GATEWAY=172.20.254.254
and /etc/sysconfig/network-scripts/ifcfg-eth5 GATEWAY=172.23.1.254
and /etc/sysconfig/network GATEWAY=172.20.254.254 - this will set the default gw

Ryan Ivey
Linux Administrator
Initial Source Solutions, LLC
www.initialss.com

So if I get this right:
- you have eth4 and eth5, both connected to the Internet
- you want locally initiated traffic to go out through eth4
- you want externally initiated traffic to go out through the same interface that the first packet for this connection came from

Try this:

1. Remove all default routes.

2. Add (once) to file /etc/iproute2/rt_tables (this gives human names to routing table numbers):

256 default_eth5
257 default_eth4

3. Add the eth5 default route to the 'default_eth5' routing table.

ip route add table default_eth5 to default via 172.23.1.254

4. Add the eth4 default route to the 'default_eth4' routing table:

ip route add table default_eth4 to default via 172.20.254.254

5. Set routing table decision rules: after consulting the main table (which has no default route), consult table default_eth5 if the packet is marked right, otherwise
consult table default_eth4. This has the effect of using the eth5 default route for marked packets, and the eth4 default route otherwise.

ip rule add prio 40000 fwmark 0x100/0x100 lookup default_eth5
ip rule add prio 40001 lookup default_eth4

6. Disable the Reverse Path Filter on eth5. Te reverse path filter is basically "if I receive a packet from interface A, but a reply to the packet would go through interface B != A, then I drop the packet". Since the determination where a reply would go is done without a packet and hence a packet mark, it may determine that a reply to a packet from eth5 would go through eth4, and the packet would be dropped.

Now "ip rule show" should say:
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
40000: from all fwmark 0x100/0x100 lookup default_eth5
40001: from all lookup default_eth4

"ip route show table main" should print only the local (no default) routed.

"ip route show table default_eth5" and "ip route show table default_eth4" should show the eth5 and eth4 default routes, respectively.

At this point, Internet access through eth4 should work. But this will still always use the eth4 default route, since no packets are being marked yet.

7. Set connection (!) mark on connections incoming to eth5. This is *not* the mark matched by the ip rule.

iptables -t mangle -A INPUT -i eth5 -j CONNMARK --or-mark 0x100

8. Set packet marks on outgoing packets belonging the the connections marked above. Note that the connection and packet mark are unrelated. The mark set here is the one matched by the ip rule.

iptables -t mangle -A OUTPUT -m connmark --mark 0x100/0x100 -j MARK --or-mark 0x100

opnsrc - it doesn't look like this worked. Did I forget something? Here are the files you told me to change.

ifcfg-eth4
DEVICE=eth4
BOOTPROTO=none
HWADDR=D00:FD:B5:FB:54
ONBOOT=yes
HOTPLUG=no
DHCP_HOSTNAME=apprpprod.oemworldwide.com
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
IPADDR=172.20.4.102
NETMASK=255.255.0.0
GATEWAY=172.20.254.254

ifcfg-eth5
DEVICE=eth5
BOOTPROTO=none
HWADDR=D00:FD:B5:FB:56
ONBOOT=yes
HOTPLUG=no
DHCP_HOSTNAME=apprpprod.oemworldwide.com
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
IPADDR=172.23.1.51
NETMASK=255.255.255.0
GATEWAY=172.23.1.254

network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=apprpprod.oemworldwide.com
GATEWAY=172.20.254.254

route
Destination Gateway Genmask Flags Metric Ref Use Iface
172.23.1.0 * 255.255.255.0 U 0 0 0 eth5
169.254.0.0 * 255.255.0.0 U 0 0 0 eth5
172.20.0.0 * 255.255.0.0 U 0 0 0 eth4
default 172.23.1.254 0.0.0.0 UG 0 0 0 eth5



ambrop7 - I will look at your suggestion next.

I'm assuming that's after a reboot or 'service network restart'.

You don't have any /etc/sysconfig/network-scripts/route-* files? I don't believe they are there by default, so I doubt you do. The main files are the ones I provided and you don't have anything missing as far as making the settings permanent.