| billias |
09-15-2018 07:20 AM |
Kubernetes DNAT and multiple Gateways (subnets)
I do experience a maybe "split routing" when IPTables is involved.
I have a System with the eth0 (enp0s3f6) and a vlan4000 ethernet
Both have different public ips (in my example I will use a private for vlan4000).
IF I use netcat and listen to a port i can connect to both IPs from remote (overseas ) location, but for some reason the DNAT redirects exist at IPTABLES do not work!.
This begins when the second default route with separate Table been added
My setup:
Quote:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet 195.001.001.001 peer 195.001.001.00232 scope global enp0s31f6
valid_lft forever preferred_lft forever
5: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
inet 10.203.0.0/12 brd 10.207.255.255 scope global weave
valid_lft forever preferred_lft forever
19: vlan4000@enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
inet 172.16.3.2/24 scope global vlan4000
valid_lft forever preferred_lft forever
|
My Roules
Quote:
0: from all lookup local
32763: from all fwmark 0x14 lookup vswitch
32764: from all to 172.16.3.0/24 lookup vswitch
32765: from 172.16.3.0/24 lookup vswitch
32766: from all lookup main
32767: from all lookup default
|
Here on this rule set I tried also to add a mark in packets.
My Iptables is a HUGE list of rules ~1500 lines.
I am going to follow a service which is on NodePort 30264 as example
Full mangle table:
Quote:
# Generated by iptables-save v1.6.1 on Sat Sep 15 14:00:01 2018
*mangle
:PREROUTING ACCEPT [3400153:5276887124]
:INPUT ACCEPT [2714280:3119195483]
:FORWARD ACCEPT [686015:2157736456]
:OUTPUT ACCEPT [1201697:1669049405]
:POSTROUTING ACCEPT [1887688:3826784901]
:WEAVE-IPSEC-IN - [0:0]
:WEAVE-IPSEC-IN-MARK - [0:0]
:WEAVE-IPSEC-OUT - [0:0]
:WEAVE-IPSEC-OUT-MARK - [0:0]
-A PREROUTING -s 172.16.3.0/24 -p tcp -m tcp --dport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A INPUT -j WEAVE-IPSEC-IN
-A OUTPUT -j WEAVE-IPSEC-OUT
-A OUTPUT -p tcp -m tcp --sport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A OUTPUT -o vlan4000 -p tcp -m tcp --sport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A WEAVE-IPSEC-IN -s 195.i/32 -d <server IP>/32 -p esp -m esp --espspi 600204388 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 19/32 -d <server IP>/32 -p esp -m esp --espspi 3480531921 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 78/32 -d <server IP>/32 -p esp -m esp --espspi 57113620 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 195./32 -d <server IP>/32 -p esp -m esp --espspi 1496250578 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 195./32 -d<server IP>/32 -p esp -m esp --espspi 218105982 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 159.6/32 -d <server IP>/32 -p esp -m esp --espspi 2842146921 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.21/32 -d <server IP>/32 -p esp -m esp --espspi 1194115493 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.218/32 -d <server IP>/32 -p esp -m esp --espspi 125481404 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.21/32 -d <server IP>/32 -p esp -m esp --espspi 2364462866 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.216/32 -d <server IP>/32 -p esp -m esp --espspi 2612186 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN-MARK -j MARK --set-xmark 0x20000/0x20000
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.j/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.i/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 78.h/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.g/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.f/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 159.e/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.d/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.c/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.b/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.a/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT-MARK -j MARK --set-xmark 0x20000/0x20000
COMMIT
|
Filter table:
Quote:
# Completed on Sat Sep 15 14:05:46 2018
# Generated by iptables-save v1.6.1 on Sat Sep 15 14:05:46 2018
*filter
:INPUT ACCEPT [13011:14415931]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [5673:7803598]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-IPSEC-IN - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-EGRESS - [0:0]
:WEAVE-NPC-EGRESS-ACCEPT - [0:0]
:WEAVE-NPC-EGRESS-CUSTOM - [0:0]
:WEAVE-NPC-EGRESS-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -i weave -j WEAVE-NPC-EGRESS
-A INPUT -j WEAVE-IPSEC-IN
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -i weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC-EGRESS
-A FORWARD -o weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -p esp -m policy --dir out --pol none -m mark --mark 0x20000/0x20000 -j DROP
|
NAT table.. A lot of rules, trying to keep only the example
We begin here
Quote:
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -j WEAVE
|
Next is: I do not know how Nodeport is accessed! i cannot find anything with -j KUBE-NODEPORTS
Quote:
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "comp1-mail/haproxy:smtp" -m tcp --dport 30264 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "comp1-mail/haproxy:smtp" -m tcp --dport 30264 -j KUBE-XLB-XSQKFO5JMTM6NPY2
|
following the KUBE-XLB-XSQKFO5JMTM6NPY2
Quote:
-A KUBE-XLB-XSQKFO5JMTM6NPY2 -s 10.192.0.0/12 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-XSQKFO5JMTM6NPY2
-A KUBE-XLB-XSQKFO5JMTM6NPY2 -m comment --comment "Balancing rule 0 for comp1-mail/haproxy:smtp" -j KUBE-SEP-D7BLSACGGFYMS44U
|
Full iptables rules: https://pastebin.com/69CZyq4a
Thanks in advance for any suggestion and even looking into this. |
