Not your average Policy-based routing question? (Help needed)

erabaddosi-2116 · 03-26-2018, 04:05 PM

Hello!

I am attempting to perform port-based policy routing, but naturally "it" is not working. Environment:

There is an incoming openvpn tunnel landing on "problem" router (AWS EC2 Linux machine). Two upstream gateways: "normal" default gateway and "target" default gateway. Port 80 traffic needs to go to "target" gateway (running squid, for the curious -- I could run it on the "problem" host, but I'd like to separate compute resources for the time being).

The default firewall policy is accept for all tables. Openvpn dumps decrypted traffic (Internet-bound) through PREROUTING, so the marking rule is successfully picking it up (per iptables -t nat -L -v -n):

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere anywhere tcp dpt:http MARK set 0x1

Later, POSTROUTING sees the mark and then MASQUERADE NATs the traffic (successfully).

iproute2 is available on this system and is configure:

"ip rule list" displays:

0: from all lookup local
32765: from all fwmark 0x1 lookup www
32766: from all lookup main
32767: from all lookup default
60000: from all fwmark 0x1 lookup www

"ip route show table www" displays:

default via 172.16.0.103 dev eth0

172.16.0.103 is the IP of the "target" gateway.

cat /proc/sys/net/ipv4/conf/*/rp_filter shows all 0.

cat /proc/sys/net/ipv4/tcp_fwmark_accept shows 1.

tcpdump -ni eth0 -xe port 80 never shows traffic leaving via the correct upstream MAC.

Is SNAT/MASQUERADE eating fwmarks? If I switch the configuration to TOS, the tcpdump successfully show TOS being marked on the packets...on the way to the wrong gateway...which is part of why I wonder if SNAT/MASQUERADE are being problematic.

Help? What add'l info can I provide?

erabaddosi-2116 · 03-30-2018, 09:30 AM

Update: Switching to ToS works. Perhaps I forgot to flush the route cache (ip route flu cache) or perhaps this was before I added the priority 60,000 rule (ip rule add fwmark 1 table www pri 60000).

Aside from the situation being resolved (for me), it is STILL strange that fwmarks are not being received by iproute2 after the PREROUTING chain. This will likely need attention again, by someone, at some point.