I'm currently experiencing odd behavior with iptables. Some virtual machines use own hypervisor as default gateway (to mitigate single point of failure), meanwhile few others use different gateway on different hypervisor. All gateways are on the same subnet.

While simple masquerade rule works with local VM - local hypervisor setup, it does double NATing otherwise.

Let's say VM has default gw 192.168.34.1 and it actually run's on hypervisor with ip 192.168.34.3
Packet will traverse NAT like this:
While I think I understand why this rule doesn't do exactly what I want, I'd like to know how to write iptables rules like:

1) If destination is in outer network and default gateway is not local, pass the packet to gateway.
2) If destination is in outer network, NAT packet and forward it.

Is it possible if packet only contains src and dst ip address?

And please don't suggest placing virtual machines "close" to it's gateway, it's not possible and even not desired right now (distributing load and logical separation of VM's across physical servers).

Can we see a packet capture with some test traffic?