After much trial, error, and research... I have managed to get a working solution by not masquerading any connections originating from the tunnel(s) of Router2 with a destination within LAN1 and then placing a static route in the router1 back to the tunnel gateway (Router2) for the tunnel ip subnet.
I am curious why I am able to traverse LAN1 out to the internet using masquerade from the tunnel, however reply packets (only from within LAN1) answering masqueraded requests that originated in the tunnel of Router1 are blocked (I say blocked since I was never able to verify if the packets were dropped or silently dropped...) Could this be bug in iptables or conntrack?
i.e. (differences in red)
The following will only work in a non-masqueraded state:
While packets traversing LAN1 out to the internet (internet pass-through) works just fine masqueraded:
excerpt from # iptables -nvL -t nat
Chain POSTROUTING (policy ACCEPT 644 packets, 54030 bytes)
pkts bytes target prot opt in out source destination
673 39221 MASQUERADE all -- * vlan1 0.0.0.0/0 0.0.0.0/0
360 18106 MASQUERADE all -- * vlan2 192.168.109.0/24 0.0.0.0/0
4 328 MASQUERADE all -- * vlan2 0.0.0.0/0 !192.168.9.0/24
Static route added to Router1
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.19.109.0 192.168.9.120 255.255.255.0 UG 0 0 0 br0