I wanted to use a server as a default gateway for its subnet, too. Unfortunately, in this configuration, return packets from the tunnel are not forwarded out to the subnet workstations.

issue:
on 192.168.1.4: ping -c1 -W1 8.8.8.8
A ping to the Internet from .4 via .2 and .114 is not returned. The response stops on its way back at .2 and is not forwarded to .4 as expected. The response arrives on tap1 but does not get sent out on eth0.

192.168.1.4 can ping: 192.168.1.2, 192.168.2.7, 192.168.2.6
192.168.1.4 can not ping: 172.31.0.114, 8.8.8.8
172.31.0.114 can ping: 8.8.8.8, 192.168.2.6, 192.168.2.7

The masquerade setup is the same on .114 and .2 so I can't figure out why .2 does not forward the responses.

Here's the setup:

=== subnet workstation 192.168.1.4:
$ ip route show
default via 192.168.1.2 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.4 metric 1

=== subnet server 192.168.1.2:
$ ip route show
default via 192.168.1.1 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1000
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev tap1 proto kernel scope link src 192.168.2.7
$ ip rule show
0: from all lookup local
32765: from all fwmark 0x3 lookup gwtable
32766: from all lookup main
32767: from all lookup default
$ ip route show table gwtable
default via 192.168.2.6 dev tap1
192.168.2.0/24 dev tap1 scope link
$ sudo iptables --table nat --list-rules
-A PREROUTING ! -d 192.168.1.2/32 -i eth0 -p icmp -j MARK --set-xmark 0x3/0xffffffff
-A PREROUTING ! -d 192.168.1.2/32 -i eth0 -p tcp -j MARK --set-xmark 0x3/0xffffffff
-A POSTROUTING -m mark --mark 0x3 -j MASQUERADE
# cat /proc/sys/net/ipv4/ip_forward
1

=== remote server .114: other end of tap1
# ip route show
default via 172.31.0.1 dev eth0
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.114
192.168.2.0/24 dev tap1 proto kernel scope link src 192.168.2.6
# iptables --list-rules; iptables --table nat --list-rules
-A FORWARD -i tap1 -j MARK --set-xmark 0x3/0xffffffff
-A POSTROUTING -m mark --mark 0x3 -j MASQUERADE
# cat /proc/sys/net/ipv4/ip_forward
1

Your routing configuration is ridiculously complex rather elaborate for such a simple network. It took me ages to figure out how the packets are supposed to flow, with all the MARKing and NAT overloading and whatnot.

1. From 192.168.0.x, packets arrive at 192.168.1.2
2. TCP/ICMP packets would match the PREROUTING rules in the nat table, sending them to the MARK target which marks them with 0x3
3. IP rules for mark 0x3 then sends them to the "gwtables" table, which routes them to 192.168.2.6
4. As they exit tap1 on the "subnet" server, they are NAT overloaded behind 192.168.2.7
5. Arriving at tap1 on the "remote" server, the packets are marked with 0x3
6. The packets are routed by the "main" table to either 172.31.0.1 (for Internet traffic) or directly to hosts on the 172.31.0.0/20 network
7. As they exit eth0, the rule matching the 0x3 mark in the POSTROUTING chain of the nat table causes them to be NAT overloaded behind 172.31.0.114
8. The router or a workstation on the 172.31.0.0/20 network receives a packet seemingly from 172.31.0.114, does its thing and sends a reply to 172.31.0.114
9. The "remote" server receives the packet, finds an entry in the conntrack and NAT tables for the packet, and does a reverse NAT
10. The reply packet, now with 192.168.2.7 as its destination address, is forwarded to the "subnet" server
11. The "subnet" server receives the packet, finds a conntrack/NAT entry, and de-NATs it
12. The "subnet" server forwards the reply packet to the original source host

As for why this doesn't work, I honestly can't say. Try running tcpdump -i tap1 on the "subnet" server and check the destination address of the reply packets from the "remote" server. If the packets look OK, check the filter table on the "subnet" server for any rules that may be blocking the traffic.

You could also get rid of the MARK and NAT rules and the "gwtable" routing table altogether, and replace them with a single static route on both servers and a general MASQUERADE rule for packets exiting eth0 on the "remote" server.