KVM server with bridge-need iptables assistance
So, I upgraded from my laptop KVM environment to a standalone 8 core Avoton server for my lab system. I had everything down perfectly on the laptop, so figured it would be an easy transition. It hasn't been.
I am setting up a lab environment for Openstack/Satellite/Ansible experiments, and need bridging to be working. I have 1 primary NIC for the server, and I have configured a second NIC as a bridged interface: ~~~ # ip addr show 2: enp0s20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 0c:c4:7a:6c:fd:fc brd ff:ff:ff:ff:ff:ff inet 192.168.1.15/24 brd 192.168.1.255 scope global enp0s20f0 valid_lft forever preferred_lft forever inet6 2602:306:bcb6:8a80:8b0a:ae12:d565:c071/64 scope global noprefixroute dynamic valid_lft 2591544sec preferred_lft 604344sec inet6 fe80::eb6d:6e14:8fe2:95ee/64 scope link valid_lft forever preferred_lft forever 3: enp0s20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000 link/ether 0c:c4:7a:6c:fd:fd brd ff:ff:ff:ff:ff:ff <snip> 19: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 0c:c4:7a:6c:fd:fd brd ff:ff:ff:ff:ff:ff inet 192.168.1.73/24 brd 192.168.1.255 scope global dynamic br0 valid_lft 74524sec preferred_lft 74524sec inet6 2602:306:bcb6:8a80:a92e:b3ae:9dfe:307c/64 scope global noprefixroute dynamic valid_lft 2591544sec preferred_lft 604344sec inet6 fe80::626a:4e3c:1c01:ed5b/64 scope link valid_lft forever preferred_lft forever ~~~ enp0s20f1 is bridged to br0. My satellite is configured to use br0. I can ping the satellite OS from my laptop, and vice-versa. I can log into the satellite server remotely from my laptop. I cannot ping out past my gateway, 192.168.1.254, and I cannot ping the KVM host from the guest, either. I think the routing on the KVM host is correct: ~~~ [root@zika disks]# ip route show default via 192.168.1.254 dev enp0s20f0 default via 192.168.1.254 dev br0 proto static metric 425 172.17.13.0/24 dev virbr1 proto kernel scope link src 172.17.13.1 linkdown 192.168.1.0/24 dev enp0s20f0 proto kernel scope link src 192.168.1.15 metric 100 192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.73 metric 425 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown ~~~ The IP is set correctly on the Satellite: ~~~ [root@satellite ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:d6:c8:0f brd ff:ff:ff:ff:ff:ff inet 192.168.1.73/24 brd 192.168.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 2602:306:bcb6:8a80:5054:ff:fed6:c80f/64 scope global dynamic valid_lft 2591974sec preferred_lft 604774sec inet6 fe80::5054:ff:fed6:c80f/64 scope link valid_lft forever preferred_lft forever ~~~ and I think the route is correct, as well: ~~~ [root@satellite ~]# ip route show default via 192.168.1.254 dev eth0 proto static metric 100 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.73 metric 100 ~~~ So I assume that there are some iptables rules that will be necessary to get this working correctly? I pose that as a question, because this is where I seem to have run past my knowledge. Could someone please point me in the right direction? |
It may be sysctl that you need to use.
|
Hi there, and thanks for replying.
If you're thinking of the ip4.forward it's set to 1 on both host and guest. |
I think I have it figured out, but I'm not sure how to resolve it, still:
Pinging a target with no answer: ~~~ [root@satellite ~]# ping redhat.com PING redhat.com (209.132.183.105) 56(84) bytes of data. ^C --- redhat.com ping statistics --- 153 packets transmitted, 0 received, 100% packet loss, time 151999ms ~~~ tcpdump on the KVM host br0 NIC shows ICMP requests, but no replies: ~~~ [root@zika disks]# tcpdump -nni enp0s20f1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s20f1, link-type EN10MB (Ethernet), capture size 262144 bytes 10:14:06.786099 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 123, length 64 10:14:07.786120 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 124, length 64 10:14:08.786086 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 125, length 64 10:14:09.786085 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 126, length 64 10:14:10.786077 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 127, length 64 10:14:11.786098 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 128, length 64 10:14:12.786113 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 129, length 64 10:14:13.786061 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 130, length 64 10:14:14.786101 IP 192.168.1.73 > 209.132.183.105: ICMP echo request, id 19767, seq 131, length 64 ~~~ If I change the NIC to the primary server IF, I see the replies coming to that interface, instead: ~~~ [root@zika disks]# tcpdump -nni enp0s20f0 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s20f0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:14:20.832846 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 137, length 64 10:14:21.832685 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 138, length 64 10:14:22.836946 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 139, length 64 10:14:23.832633 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 140, length 64 10:14:24.832732 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 141, length 64 10:14:25.832629 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 142, length 64 10:14:26.834027 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 143, length 64 10:14:27.832894 IP 209.132.183.105 > 192.168.1.73: ICMP echo reply, id 19767, seq 144, length 64 ~~~ Routing and iptables are kind of weak for me. Does anyone know why I am seeing the replies on the wrong wire? |
If I was in your situation I would get out a piece of paper and pencil and draw my environment out. Another one I do is write a bug report. Not one that I will file but just go through all the steps as if I was writing a real bug report. Often either one of those will make the problem come out and reveal itself.
|
I know I'm onto the issue, but still having a hard time working through it.
It seems like policy-based routing might be necessary here? Here's what I've been trying, with no luck: ~~~ [root@zika disks]# ip rule add iif enp0s20f0 table 1 [root@zika disks]# ip rule add iif br0 table 2 [root@zika disks]# ip rule add from 192.168.1.15 table 1 [root@zika disks]# ip rule add from 192.168.1.73 table 2 [root@zika disks]# ip route show default via 192.168.1.254 dev enp0s20f0 proto static metric 100 default via 192.168.1.254 dev br0 proto static metric 425 172.17.13.0/24 dev virbr1 proto kernel scope link src 172.17.13.1 linkdown 192.168.1.0/24 dev enp0s20f0 proto kernel scope link src 192.168.1.15 metric 100 192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.73 metric 425 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [root@zika disks]# ip route show table 1 default via 192.168.1.254 dev enp0s20f0 192.168.1.0/24 dev enp0s20f0 scope link [root@zika disks]# ip route show table 2 default via 192.168.1.254 dev br0 192.168.1.0/24 dev br0 scope link ~~~ The only time things were working was when I accidentally broke the main routing rules by using 'ip rule flush'. As soon as I did that, the primary IF was no longer communicating, but the secondary was. As soon as I restored the MAIN routing rule: 'ip rule add from all lookup main pref 32768' The secondary IF was again no longer receiving traffic. |
Quote:
I tried a Red Hat doc: How can I route network traffic such that the packets go out via the same interface they came in? https://access.redhat.com/solutions/19596 But that didn't work. |
Code:
| Lab Server Satellite VM| br0 is 192.168.1.73 |
All times are GMT -5. The time now is 12:56 PM. |