[SOLVED] complex routing problem using ssh with Wireguard and iptables

pisti · 03-11-2022, 11:39 PM

i am trying to solve a, for my taste at least, complex real-life-like routing problem which may be daily business for professionals at companies and large organizations but not for me amateur who tries to implement a safe path for accessing a startup's computing environment from the public internet.

the scenario is the following : a bunch of CLIENT computers on the internet need to access in a virtual private server (VPS) farm a bunch of VPS HOSTs through the startup VPS PORTAL. literally all machines have some form of public IP address - but the VPSs should only be accessable via PORTAL. all VPSs run Slackware Linux 15.0. Wireguard is the VPN protocol used between all machines. the VPS firewalls use iptables. all traffic between CLIENTs and VPSs use ssh or derivates and tigerVNC for GUI based works. i am not sure how this network scenario is called though i guess it must be a form of classic NAT setup.

with the schema below nor PING nor SSH works from ClientA to HostA. but when i type :

Code:

iptables -P FORWARD ACCEPT

then at least PING begins to work, from ClientA to PORTAL, and to HostA :

Code:

ping 10.33.1.1
PING 10.33.1.1 (10.33.1.1) 56(84) bytes of data.
64 bytes from 10.33.1.1: icmp_seq=1 ttl=64 time=3.31 ms
64 bytes from 10.33.1.1: icmp_seq=2 ttl=64 time=1.15 ms
64 bytes from 10.33.1.1: icmp_seq=3 ttl=64 time=1.04 ms
64 bytes from 10.33.1.1: icmp_seq=4 ttl=64 time=4.22 ms
^C
--- 10.33.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.044/2.430/4.221/1.371 ms

ping 10.66.1.1
PING 10.66.1.1 (10.66.1.1) 56(84) bytes of data.
64 bytes from 10.66.1.1: icmp_seq=1 ttl=63 time=13.0 ms
64 bytes from 10.66.1.1: icmp_seq=2 ttl=63 time=11.3 ms
64 bytes from 10.66.1.1: icmp_seq=3 ttl=63 time=8.23 ms
64 bytes from 10.66.1.1: icmp_seq=4 ttl=63 time=9.27 ms

while SSH still refuses to work :

Code:

ssh somebody@10.66.1.1
ssh: connect to host 10.66.1.1 port 22: Connection refused

schema map :

Code:

ClientA : IP=1.2.3.4 ,     wgIP=10.33.1.2 , port=11111 , ClientA.key , ClientA.pub

Portal :  IP=11.22.33.44 , wgIP=10.33.1.1 , port=11111 , Portal1.key , Portal1.pub
  "       IP=11.22.33.44 , wgIP=10.66.1.2 , port=22222 , Portal2.key , Portal2.pub

HostA   : IP=22.33.44.55 , wgIP=10.66.1.1 , port=22222 , HostA.key ,   HostA.pub

ClientA : wg_ClientA_to_Portal.conf :

Code:

[Interface]
Address = 10.33.1.2/24
ListenPort = 11111
PrivateKey = ClientA.key

[Peer]
PublicKey = Portal1.pub
AllowedIPs = 10.33.1.1/32,10.66.1.0/24
Endpoint = 11.22.33.44:11111

ClientA : sshd_conf :

Code:

ListenAddress 10.33.1.2

ClientA : firewall.build :

Code:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -m comment --comment "Allow loopback connections" -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -p icmp -m comment --comment "Allow Ping to work as expected" -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -m multiport --destination-ports 11111 -j ACCEPT
iptables -A INPUT -p udp -m udp -m multiport --destination-ports 11111 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -m multiport --destination-ports 5900:6000 -j ACCEPT
iptables -A INPUT -p udp -m udp -m multiport --destination-ports 5900:6000 -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -s 10.33.1.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 10.33.1.0/24 --dport 873 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

Portal : wg_Portal_to_ClientA.conf :

Code:

[Interface]
Address = 10.33.1.1/24
ListenPort = 11111
PrivateKey = Portal1.key

[Peer]
PublicKey = CLientA.pub
AllowedIPs = 10.33.1.2/32
Endpoint = 1.2.3.4:11111

Portal : wg_Portal_to_HostA.conf

Code:

[Interface]
Address = 10.66.1.2/24
ListenPort = 22222
PrivateKey = Portal2.key

[Peer]
PublicKey = HostA.pub
AllowedIPs = 10.66.1.1/32
Endpoint = 22.33.44.55:22222

Portal : firewall.build :

Code:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -m comment --comment "Allow loopback connections" -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -p icmp -m comment --comment "Allow Ping to work as expected" -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -m multiport --destination-ports 11111,22222 -j ACCEPT
iptables -A INPUT -p udp -m udp -m multiport --destination-ports 11111,22222 -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -s 10.33.1.2/32 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -s 10.66.1.1/32 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING 1 -s 10.33.0.0/16 -o wg_Portal_to_HostA -j MASQUERADE
iptables -t nat -I POSTROUTING 1 -s 10.33.0.0/16 -d 10.66.1.1/32 -j MASQUERADE
iptables -I INPUT 1 -s 10.33.0.0/16 -j ACCEPT
iptables -I FORWARD 1 -i wg_Portal_to_HostA -o eth0 -j ACCEPT
iptables -I FORWARD 1 -s 10.33.0.0/16 -d 10.66.1.1/32 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

HostA : wg_HostA_to_Portal.conf

Code:

[Interface]
Address = 10.66.1.1/24
ListenPort = 22222
PrivateKey = HostA.key

[Peer]
PublicKey = Portal2.pub
AllowedIPs = 10.66.1.2/32,10.33.0.0/16
Endpoint = 11.22.33.44:22222

HostA : sshd_conf :

Code:

ListenAddress 10.66.1.1

HostA : firewall.build :

Code:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -m comment --comment "Allow loopback connections" -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -p icmp -m comment --comment "Allow Ping to work as expected" -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -m multiport --destination-ports 22222 -j ACCEPT
iptables -A INPUT -p udp -m udp -m multiport --destination-ports 22222 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -m multiport --destination-ports 5900:6000 -j ACCEPT
iptables -A INPUT -p udp -m udp -m multiport --destination-ports 5900:6000 -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -s 10.66.1.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 10.66.1.0/24 --dport 873 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

i hope i didn't introduce any typos in this schema here...

what do i do wrong ? what needs to be changed in PORTAL's iptables commands so that users can jump with ssh via PORTAL effortlessly from ClientA to HostA ?

Code:

iptables -P FORWARD ACCEPT

shouldn't be an option, right ?

pisti · 03-12-2022, 09:05 AM

ADDENDUM : now even PING broke, beside SSH, after flushing the PORTAL's firewall and restarting its unchanged firewall.build service...which is a bit puzzling as it was consistently working last night...sigh...

monitoring incoming traffic doesn't show one iota of sign of life :

Code:

tcpdump -ttttni eth0 'udp port 22222'
tcpdump -ttttni eth0 'tcp port 22222'

...which, as said, was fine till this morning.

i assume the network's middleman, the PORTAL's iptables, are the culprits here - what do people think ?

beka · 03-13-2022, 05:48 AM

Hi pisti,

While I am not very familiar with Wireguard, some basic factors don't seem to be right. The ip range for the VPN network used by all the machines need to be on the same subnet. You would have one Wireguard VPN server (I presume it would be the startup PORTAL) machine, and the HOST and CLIENT machines will all be Wireguard clients. If you can give us the operating systems of the different machines, maybe it would help to point you to instructions on how to do the configurations.

Best

pisti · 03-13-2022, 10:06 AM

you are entirely right, dear béka, in that using one single shared subnet is the easiest and likely most logical solution for this scenario. the OS question is perhaps less relevant, as the clients, be that Mac, Win or Lnx, only need Wireguard properly configured, and otherwise just some shell terminal.

the thorny place is the PORTAL's wg0.conf file. i just found two sources with information relevant to the problem discussed here which i am going to study tonight, a first one, Pro Custodibus, with many excellent examples :

www.procustodibus.com : wireguard hub-and-spoke

and a second one for read-up :

https://docs.sweeting.me/s/wireguard

more about all this tomorrow...

beka · 03-13-2022, 10:28 AM

Thumbs up!

pisti · 03-16-2022, 11:29 AM

ok, things are done, problem [SOLVED]. i luckily stumbled upon those truly excellent Pro Custodibus documents that helped me immensely and surely cut short my R&D time by days. i strictly followed their instructions for the HUB & SPOKE network topology type :

hub & spoke
hub & spoke configuration
hub & spoke with iptables

one just needs to pay great attention at machine boot time to start the necessary services in a well concerted fashion ! the fortunately script-based Slackware boot procedures make this a somewhat transparent undertaking. in my /etc/rc.d/rc.local i do :

Code:

1) begin with network interfaces, for example start the regular simple Wireguard config files
2) thereafter load the iptables commands with your /etc/firewall.rules file
3) only then start the hub & spoke Wireguard config file (which include again a bunch of iptables commands)
4) last start your /etc/rc.d/rc.sshd service(s)

that's it. i will not display more of my scripts because i really just copy/pasted from these aforementioned ProCustodibus pages - that should do it for here.