LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 09-15-2018, 07:20 AM   #1
billias
LQ Newbie
 
Registered: Sep 2018
Posts: 1

Rep: Reputation: Disabled
Kubernetes DNAT and multiple Gateways (subnets)


I do experience a maybe "split routing" when IPTables is involved.

I have a System with the eth0 (enp0s3f6) and a vlan4000 ethernet

Both have different public ips (in my example I will use a private for vlan4000).
IF I use netcat and listen to a port i can connect to both IPs from remote (overseas ) location, but for some reason the DNAT redirects exist at IPTABLES do not work!.

This begins when the second default route with separate Table been added

My setup:
Quote:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet 195.001.001.001 peer 195.001.001.00232 scope global enp0s31f6
valid_lft forever preferred_lft forever
5: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
inet 10.203.0.0/12 brd 10.207.255.255 scope global weave
valid_lft forever preferred_lft forever
19: vlan4000@enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
inet 172.16.3.2/24 scope global vlan4000
valid_lft forever preferred_lft forever
My Roules

Quote:
0: from all lookup local
32763: from all fwmark 0x14 lookup vswitch
32764: from all to 172.16.3.0/24 lookup vswitch
32765: from 172.16.3.0/24 lookup vswitch
32766: from all lookup main
32767: from all lookup default
Here on this rule set I tried also to add a mark in packets.

My Iptables is a HUGE list of rules ~1500 lines.
I am going to follow a service which is on NodePort 30264 as example

Full mangle table:
Quote:
# Generated by iptables-save v1.6.1 on Sat Sep 15 14:00:01 2018
*mangle
:PREROUTING ACCEPT [3400153:5276887124]
:INPUT ACCEPT [2714280:3119195483]
:FORWARD ACCEPT [686015:2157736456]
:OUTPUT ACCEPT [1201697:1669049405]
:POSTROUTING ACCEPT [1887688:3826784901]
:WEAVE-IPSEC-IN - [0:0]
:WEAVE-IPSEC-IN-MARK - [0:0]
:WEAVE-IPSEC-OUT - [0:0]
:WEAVE-IPSEC-OUT-MARK - [0:0]
-A PREROUTING -s 172.16.3.0/24 -p tcp -m tcp --dport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A INPUT -j WEAVE-IPSEC-IN
-A OUTPUT -j WEAVE-IPSEC-OUT
-A OUTPUT -p tcp -m tcp --sport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A OUTPUT -o vlan4000 -p tcp -m tcp --sport 30264 -j MARK --set-xmark 0x14/0xffffffff
-A WEAVE-IPSEC-IN -s 195.i/32 -d <server IP>/32 -p esp -m esp --espspi 600204388 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 19/32 -d <server IP>/32 -p esp -m esp --espspi 3480531921 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 78/32 -d <server IP>/32 -p esp -m esp --espspi 57113620 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 195./32 -d <server IP>/32 -p esp -m esp --espspi 1496250578 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 195./32 -d<server IP>/32 -p esp -m esp --espspi 218105982 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 159.6/32 -d <server IP>/32 -p esp -m esp --espspi 2842146921 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.21/32 -d <server IP>/32 -p esp -m esp --espspi 1194115493 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.218/32 -d <server IP>/32 -p esp -m esp --espspi 125481404 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.21/32 -d <server IP>/32 -p esp -m esp --espspi 2364462866 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 95.216/32 -d <server IP>/32 -p esp -m esp --espspi 2612186 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN-MARK -j MARK --set-xmark 0x20000/0x20000
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.j/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.i/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 78.h/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.g/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 195.f/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 159.e/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.d/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.c/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.b/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s <server IP>/32 -d 95.a/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT-MARK -j MARK --set-xmark 0x20000/0x20000
COMMIT

Filter table:
Quote:
# Completed on Sat Sep 15 14:05:46 2018
# Generated by iptables-save v1.6.1 on Sat Sep 15 14:05:46 2018
*filter
:INPUT ACCEPT [13011:14415931]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [5673:7803598]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-IPSEC-IN - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-EGRESS - [0:0]
:WEAVE-NPC-EGRESS-ACCEPT - [0:0]
:WEAVE-NPC-EGRESS-CUSTOM - [0:0]
:WEAVE-NPC-EGRESS-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -i weave -j WEAVE-NPC-EGRESS
-A INPUT -j WEAVE-IPSEC-IN
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -i weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC-EGRESS
-A FORWARD -o weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -p esp -m policy --dir out --pol none -m mark --mark 0x20000/0x20000 -j DROP
NAT table.. A lot of rules, trying to keep only the example
We begin here
Quote:
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -j WEAVE
Next is: I do not know how Nodeport is accessed! i cannot find anything with -j KUBE-NODEPORTS
Quote:
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "comp1-mail/haproxy:smtp" -m tcp --dport 30264 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "comp1-mail/haproxy:smtp" -m tcp --dport 30264 -j KUBE-XLB-XSQKFO5JMTM6NPY2
following the KUBE-XLB-XSQKFO5JMTM6NPY2

Quote:
-A KUBE-XLB-XSQKFO5JMTM6NPY2 -s 10.192.0.0/12 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-XSQKFO5JMTM6NPY2
-A KUBE-XLB-XSQKFO5JMTM6NPY2 -m comment --comment "Balancing rule 0 for comp1-mail/haproxy:smtp" -j KUBE-SEP-D7BLSACGGFYMS44U

Full iptables rules: https://pastebin.com/69CZyq4a

Thanks in advance for any suggestion and even looking into this.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] multiple lan - multiple gateways - squid vagkouts Linux - Networking 3 06-27-2017 12:37 AM
need to force seperate subnets out via different gateways on debian 6.0.1 iptables Zagaz Linux - Networking 3 12-26-2012 05:13 AM
Question regarding 2 gateways one proxy and two subnets ? bakke Linux - Networking 1 09-11-2012 08:26 AM
Red Hat 7.3 and multiple gateways on multiple interfaces bluefmc Linux - Networking 2 11-19-2004 05:01 PM
multiple ips, multiple gateways, one interface drpixel Linux - Networking 6 12-04-2002 12:56 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 08:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration