iptables - Inconsistent behavior between similar port forwarding rules

Thoom · 12-11-2019, 03:24 PM

I'm trying to set up a cloud server as a reverse proxy for my home NAS, which will soon be behind a firewall that doesn't allow inbound connections. I've got the NAS connecting to the server via OpenVPN, and am trying to set up port forwarding rules on the server.

The server is running Ubuntu 18.04. net.ipv4.ip_forward is enabled.

I have the following section set up in /etc/ufw/before.rules

Code:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

-A PREROUTING -d $SERVER_IP -m tcp -p tcp --dport 2022 -j DNAT --to-destination 10.8.0.2:22
-A PREROUTING -d $SERVER_IP -m tcp -p tcp --dport 80 -j DNAT --to-destination 10.8.0.2
-A PREROUTING -d $SERVER_IP -m tcp -p tcp --dport 443 -j DNAT --to-destination 10.8.0.2
-A PREROUTING -d $SERVER_IP -m tcp -p tcp --dport 32400 -j DNAT --to-destination 10.8.0.2

-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
-A POSTROUTING --out-interface tun0 -j MASQUERADE

COMMIT

I also have corresponding rules for port opening in /etc/ufw/user.rules:

Code:


### tuple ### allow tcp 80 0.0.0.0/0 any 0.0.0.0/0 in
-A ufw-user-input -p tcp --dport 80 -j ACCEPT

### tuple ### allow tcp 443 0.0.0.0/0 any 0.0.0.0/0 in
-A ufw-user-input -p tcp --dport 443 -j ACCEPT

### tuple ### allow tcp 2022 0.0.0.0/0 any 0.0.0.0/0 in
-A ufw-user-input -p tcp --dport 2022 -j ACCEPT

### tuple ### allow tcp 32400 0.0.0.0/0 any 0.0.0.0/0 in
-A ufw-user-input -p tcp --dport 32400 -j ACCEPT

Traffic bound for ports 80/443 gets forwarded correctly. Traffic for 2022 and 32400 gets a "connection refused" error, e.g. if I execute (from a client on a different network):

Code:

> curl -k https://$SERVER_IP:32400
curl: (7) Failed to connect to $SERVER_IP port 32400: Connection refused

But I know the service is running at port 32400 on the NAS, because if I execute (from the server):

Code:

> curl -k https://10.8.0.2:32400
<html>...

I get a correct response.

Another strange thing I noticed, is that when I run "iptables -t nat -L PREROUTING -v", I get the following:

Code:

Chain PREROUTING (policy ACCEPT 293 packets, 18812 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  any    any     anywhere             $SERVER_IP        tcp dpt:2022 to:10.8.0.2:22
    2   128 DNAT       tcp  --  any    any     anywhere             $HOSTNAME         tcp dpt:http to:10.8.0.2
    0     0 DNAT       tcp  --  any    any     anywhere             $HOSTNAME         tcp dpt:https to:10.8.0.2
    0     0 DNAT       tcp  --  any    any     anywhere             $SERVER_IP        tcp dpt:32400 to:10.8.0.2

On the working rules, the destination column shows the hostname of the server machine, whereas the other rules show its IP.

Another minor frustration is that for a few minutes earlier, the 2022 forward *was* working, but it broke after a reboot, and I can't for the life of me remember what rules might have changed in the reboot.

Thoom · 12-11-2019, 06:46 PM

Looking deeper into this, I added some tracing.

Code:

iptables -t raw -A PREROUTING -p tcp --dport 32400 -j TRACE

And when I send a connection request on that port, I see in the logs that the nat:PREROUTING rule for port 32400 simply isn't matching,

Code:

Dec 12 00:29:50 $HOSTNAME kernel: [  484.138325] TRACE: raw:PREROUTING:policy:3 IN=eth0 OUT= MAC=$MAC_ADDR SRC=$CLIENT_IP DST=$SERVER_IP LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=64643 DPT=32400 SEQ=3487167416 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B4010303060101080A0C377E8C0000000004020000)
Dec 12 00:29:50 $HOSTNAME kernel: [  484.138385] TRACE: nat:PREROUTING:policy:5 IN=eth0 OUT= MAC=$MAC_ADDR SRC=$CLIENT_IP DST=$SERVER_IP LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=64643 DPT=32400 SEQ=3487167416 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B4010303060101080A0C377E8C0000000004020000)
Dec 12 00:29:50 $HOSTNAME kernel: [  484.138421] TRACE: filter:INPUT:rule:1 IN=eth0 OUT= MAC=$MAC_ADDR SRC=$CLIENT_IP DST=$SERVER_IP LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=64643 DPT=32400 SEQ=3487167416 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B4010303060101080A0C377E8C0000000004020000)

Which I find completely bizarre, because it's literally copy&pasted from the correctly working port 80 rule, just with 80 changed to 32400.

What the heck is going on here?

Thoom · 12-11-2019, 07:06 PM

Removing the -d $SERVER_IP clause from the broken rules made them work. But why? Why does it work with some ports and not others?

What's the etiquette here for when I've technically fixed my problem, but really want to know why the fix worked? Should I still mark the thread [SOLVED]?