Red Hat 7 on Dell server. Can't get p2p1 and em1 to be connected to different subnets at the same time.

matthewst · 04-21-2016, 01:37 PM

I can bring up either one and ssh and browse to it's assigned address as long as the other interface is down. If I attempt to bring the both up I can't connect to either one. Just one at a time. Both are on different subnets and have different default gateways.

em0 = 192.168.2.4
p2p1 = 67.xxx.xxx.10

netstat -rn

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 67.xxx.xxx.9 0.0.0.0 UG 0 0 0 p2p1
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 em2
67.xxx.xxx.8 0.0.0.0 255.255.255.252 U 0 0 0 p2p1
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 em2
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

ip route show

default via 67.xxx.xxx.9 dev p2p1 proto static metric 100
default via 192.168.2.1 dev em2 proto static metric 101
67.xxx.xxx.8/30 dev p2p1 proto kernel scope link src 67.xxx.xxx.10 metric 100
192.168.2.0/24 dev em2 proto kernel scope link src 192.168.2.4 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

etc/sysconfig/network

#Created by anaconda
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=yourmom.myhouse.local
ADDRESS0=192.168.2.4
NETMASK0=255.255.255.0
GATEWAY0=192.168.2.1
ADDRESS1=67.xxx.xxx.10
NETMASK1=255.255.255.252
GATEWAY1=67.xxx.xxx.9

Has anyone here ever been able to get something like this to work?

suicidaleggroll · 04-21-2016, 02:49 PM

You can't have two default gateways, that doesn't make sense. When you access a machine on 192.168.2.0/24, it will go through em2. When you access a machine on 67.xxx.xxx.8/30, it will go through p2p1. The "default" is the one it chooses when you access a machine that isn't on either of those subnets. There can only be one default.

The servers in my office are all on two subnets. A 192.168.1.0/24 1 GbE that connects them to the rest of the network, and a 192.168.2.0/24 10 GbE that only connects them to each other. The 192.168.1.0/24 interface is set as the default, so all outbound traffic to the wide world goes out of that interface (the 10 GbE network has no access to the outside world).

Here's the config for one of the CentOS 7 machines:

Code:

$ cat /etc/sysconfig/network-scripts/ifcfg-ens1f0 
DEVICE=ens1f0
HWADDR=blah
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=192.168.1.144
PREFIX=24
GATEWAY=192.168.1.1
DNS1=192.168.1.2
DNS2=192.168.1.3
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

$ cat /etc/sysconfig/network-scripts/ifcfg-ens1f1
DEVICE=ens1f1
HWADDR=blah
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=192.168.2.144
PREFIX=24
GATEWAY=192.168.2.1
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

$ ip route show
default via 192.168.1.1 dev ens1f0 
192.168.1.0/24 dev ens1f0  proto kernel  scope link  src 192.168.1.144 
192.168.2.0/24 dev ens1f1  proto kernel  scope link  src 192.168.2.144

matthewst · 04-22-2016, 02:28 PM

I ended up using the following routing table:

echo "3 public" >> /etc/iproute2/rt_tables
ip route add 67.xxx.xxx.0/16 dev p2p1 src 67.xxx.xxx.10 table public
ip route add default via 67.xxx.xxx.9 dev p2p1 table public
ip rule add from 67.xxx.xxx.10/32 table public
ip rule add to 67.xxx.xxx.10/32 table public

I have to enter that after each reboot. Does anyone know how to make that permanent?...other than using the script here: http://brainscraps.wikia.com/wiki/Se...ork_Interfaces.