LinuxQuestions.org - policy routing problem with fwmark

Here's what I think. The outbound packets from the PC with the address 172.16.78.138 are being routed properly, and replies are coming in. The tcpdump session proved that. I'd say that proves the policy routing is actually working; the outbound packet is being processed by the right routing table.

The question is, why isn't the reply packet being routed back to 172.16.78.138? It can't be policy routing, as the packet doesn't match the criteria in the iptables fwmark rule, and shouldn't be subject to policy routing anyway.

The way I see it, there are just three mechanisms that could prevent the packet from reaching 172.16.78.138:

Routing
Reverse path verification
The iptables firewall

You've ruled out (1) and (2), which leaves (3).

I think you may be onto something when you suspect a kernel/iproute2/firewall bug. If the policy routing somehow conflicts with the iptables state engine, the reply packet may not match the state rules for ESTABLISHED or RELATED traffic.

I believe it's possible to troubleshoot this further. How about adding a temporary rule to the top of the FORWARD chain in the filter table, like this:

Code:

iptables -I FORWARD 1 -i eth0.666 -s 8.8.8.8 -d 172.16.78.138 -p icmp -j ACCEPT

That rule would match ANY icmp packet from 8.8.8.8 to 172.16.78.138 arriving at eth0.666, regardless of whether it's a reply or not. The packet would have to reach the routing engine, no matter how erroneous the entries in the firewall state engine might be.

Could you try the ping test again with that rule in place? (There should be no risk of it disturbing network traffic, and it can be easily removed afterwards with iptables -D FORWARD 1.)