KVM server with bridge-need iptables assistance

GaWdLy · 07-29-2016, 07:46 PM

So, I upgraded from my laptop KVM environment to a standalone 8 core Avoton server for my lab system. I had everything down perfectly on the laptop, so figured it would be an easy transition. It hasn't been.

I am setting up a lab environment for Openstack/Satellite/Ansible experiments, and need bridging to be working.

I have 1 primary NIC for the server, and I have configured a second NIC as a bridged interface:

~~~
# ip addr show

2: enp0s20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 0c:c4:7a:6c:fd:fc brd ff:ff:ff:ff:ff:ff
inet 192.168.1.15/24 brd 192.168.1.255 scope global enp0s20f0
valid_lft forever preferred_lft forever
inet6 2602:306:bcb6:8a80:8b0a:ae12:d565:c071/64 scope global noprefixroute dynamic
valid_lft 2591544sec preferred_lft 604344sec
inet6 fe80::eb6d:6e14:8fe2:95ee/64 scope link
valid_lft forever preferred_lft forever
3: enp0s20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 0c:c4:7a:6c:fd:fd brd ff:ff:ff:ff:ff:ff
<snip>
19: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:6c:fd:fd brd ff:ff:ff:ff:ff:ff
inet 192.168.1.73/24 brd 192.168.1.255 scope global dynamic br0
valid_lft 74524sec preferred_lft 74524sec
inet6 2602:306:bcb6:8a80:a92e:b3ae:9dfe:307c/64 scope global noprefixroute dynamic
valid_lft 2591544sec preferred_lft 604344sec
inet6 fe80::626a:4e3c:1c01:ed5b/64 scope link
valid_lft forever preferred_lft forever
~~~

enp0s20f1 is bridged to br0.

My satellite is configured to use br0. I can ping the satellite OS from my laptop, and vice-versa. I can log into the satellite server remotely from my laptop.

I cannot ping out past my gateway, 192.168.1.254, and I cannot ping the KVM host from the guest, either.

I think the routing on the KVM host is correct:

~~~
[root@zika disks]# ip route show
default via 192.168.1.254 dev enp0s20f0
default via 192.168.1.254 dev br0 proto static metric 425
172.17.13.0/24 dev virbr1 proto kernel scope link src 172.17.13.1 linkdown
192.168.1.0/24 dev enp0s20f0 proto kernel scope link src 192.168.1.15 metric 100
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.73 metric 425
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
~~~

The IP is set correctly on the Satellite:

~~~
[root@satellite ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:d6:c8:0f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.73/24 brd 192.168.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 2602:306:bcb6:8a80:5054:ff:fed6:c80f/64 scope global dynamic
valid_lft 2591974sec preferred_lft 604774sec
inet6 fe80::5054:ff:fed6:c80f/64 scope link
valid_lft forever preferred_lft forever
~~~

and I think the route is correct, as well:

~~~
[root@satellite ~]# ip route show
default via 192.168.1.254 dev eth0 proto static metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.73 metric 100
~~~

So I assume that there are some iptables rules that will be necessary to get this working correctly? I pose that as a question, because this is where I seem to have run past my knowledge.

Could someone please point me in the right direction?

Rinndalir · 07-29-2016, 11:20 PM

It may be sysctl that you need to use.

GaWdLy · 07-29-2016, 11:26 PM

Hi there, and thanks for replying.

If you're thinking of the ip4.forward it's set to 1 on both host and guest.

GaWdLy · 07-30-2016, 12:17 PM

I think I have it figured out, but I'm not sure how to resolve it, still:

Pinging a target with no answer:
~~~
[root@satellite ~]# ping redhat.com
PING redhat.com (209.132.183.105) 56(84) bytes of data.
^C
--- redhat.com ping statistics ---
153 packets transmitted, 0 received, 100% packet loss, time 151999ms
~~~

tcpdump on the KVM host br0 NIC shows ICMP requests, but no replies:
~~~
[root@zika disks]# tcpdump -nni enp0s20f1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s20f1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:14:06.786099 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 123, length 64
10:14:07.786120 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 124, length 64
10:14:08.786086 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 125, length 64
10:14:09.786085 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 126, length 64
10:14:10.786077 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 127, length 64
10:14:11.786098 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 128, length 64
10:14:12.786113 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 129, length 64
10:14:13.786061 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 130, length 64
10:14:14.786101 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 131, length 64
~~~

If I change the NIC to the primary server IF, I see the replies coming to that interface, instead:
~~~
[root@zika disks]# tcpdump -nni enp0s20f0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s20f0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:14:20.832846 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 137, length 64
10:14:21.832685 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 138, length 64
10:14:22.836946 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 139, length 64
10:14:23.832633 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 140, length 64
10:14:24.832732 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 141, length 64
10:14:25.832629 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 142, length 64
10:14:26.834027 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 143, length 64
10:14:27.832894 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 144, length 64
~~~

Routing and iptables are kind of weak for me. Does anyone know why I am seeing the replies on the wrong wire?

Rinndalir · 07-30-2016, 01:35 PM

If I was in your situation I would get out a piece of paper and pencil and draw my environment out. Another one I do is write a bug report. Not one that I will file but just go through all the steps as if I was writing a real bug report. Often either one of those will make the problem come out and reveal itself.

GaWdLy · 07-30-2016, 02:07 PM

I know I'm onto the issue, but still having a hard time working through it.

It seems like policy-based routing might be necessary here?

Here's what I've been trying, with no luck:

~~~
[root@zika disks]# ip rule add iif enp0s20f0 table 1
[root@zika disks]# ip rule add iif br0 table 2
[root@zika disks]# ip rule add from 192.168.1.15 table 1
[root@zika disks]# ip rule add from 192.168.1.73 table 2
[root@zika disks]# ip route show
default via 192.168.1.254 dev enp0s20f0 proto static metric 100
default via 192.168.1.254 dev br0 proto static metric 425
172.17.13.0/24 dev virbr1 proto kernel scope link src 172.17.13.1 linkdown
192.168.1.0/24 dev enp0s20f0 proto kernel scope link src 192.168.1.15 metric 100
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.73 metric 425
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
[root@zika disks]# ip route show table 1
default via 192.168.1.254 dev enp0s20f0
192.168.1.0/24 dev enp0s20f0 scope link
[root@zika disks]# ip route show table 2
default via 192.168.1.254 dev br0
192.168.1.0/24 dev br0 scope link
~~~

The only time things were working was when I accidentally broke the main routing rules by using 'ip rule flush'. As soon as I did that, the primary IF was no longer communicating, but the secondary was.

As soon as I restored the MAIN routing rule: 'ip rule add from all lookup main pref 32768'

The secondary IF was again no longer receiving traffic.

GaWdLy · 07-30-2016, 02:20 PM

Quote:

Originally Posted by Rinndalir

If I was in your situation I would get out a piece of paper and pencil and draw my environment out. Another one I do is write a bug report. Not one that I will file but just go through all the steps as if I was writing a real bug report. Often either one of those will make the problem come out and reveal itself.

Mercifully, I don't think my environment is that complicated at this point. It really is as simple as 2 interfaces on the same network, using the same gateway/default. I just don't know how to make that work, I don't think.

I tried a Red Hat doc: How can I route network traffic such that the packets go out via the same interface they came in? https://access.redhat.com/solutions/19596

But that didn't work.

GaWdLy · 07-30-2016, 04:37 PM

Code:

   | Lab Server     Satellite VM|
   |                       |    |
   |enp0s20f0          enp0s20f1|
   |(primary)           (br0)   |
   |____|__________________|____|
        |                  |
        |                  |
        |_____[SWITCH]_____|
                 |
                 |
              [GATEWAY]
                 |
                 |
              [Internet]

enp0s20f0 is 192.168.1.15
br0 is 192.168.1.73