Hello,
I tried to set up policy based routing on my box in order to force specific traffic through a vpn tunnel. This seems to work quite well, however tcpdump tells me that some packets are leaking over the default interface what seems a security risk to me. I couldn't solve the problem so that is why I am asking here now
At first some information to my network setup:
Devices:
venet0: 78.46.xxx.xxx (my main interface, it's inside an OpenVZ container)
tun0: 1.2.124.147
OpenVPN Endpoint: 213.232.208.199
# ip rule
0: from all lookup local
32762: from all fwmark 0x1 lookup vpn.out
32763: from all lookup main
32764: from all lookup main
32766: from all lookup main
# ip route show table main
192.0.2.1 dev venet0 scope link
213.232.208.199 via 192.0.2.1 dev venet0
1.2.124.0/24 dev tun0 proto kernel scope link src 1.2.124.147
1.0.0.0/8 via 1.2.124.1 dev tun0
default via 192.0.2.1 dev venet0
# ip route show table vpn.out
default via 1.2.124.1 dev tun0
This is my script for setting up the packet marking:
Code:
#!/bin/sh
./flush.sh
ip rule add fwmark 1 lookup vpn.out
ip route add 213.232.208.199/32 via 192.0.2.1 dev venet0
ip route add default dev tun0 via 1.2.124.1 table vpn.out
ip route add 1.0.0.0/8 via 1.2.124.1 dev tun0
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
iptables -t mangle -N TUNMARK
iptables -t mangle -A TUNMARK -j MARK --set-mark 1
iptables -t mangle -A TUNMARK -j CONNMARK --save-mark
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -p all -m state --state NEW -m owner --uid-owner 13 -j TUNMARK
iptables -t mangle -A OUTPUT -p all -m state --state NEW -m owner --uid-owner 107 -j TUNMARK
iptables -t mangle -A OUTPUT -p all -m state --state NEW -m owner --uid-owner 108 -j TUNMARK
iptables -t mangle -A OUTPUT -p all -m state --state NEW -m owner --uid-owner 1004 -j TUNMARK
iptables -t mangle -A OUTPUT -p all -m state --state ESTABLISHED,RELATED -m owner --uid-owner 13 -j RESTOREMARK
iptables -t mangle -A OUTPUT -p all -m state --state ESTABLISHED,RELATED -m owner --uid-owner 107 -j RESTOREMARK
iptables -t mangle -A OUTPUT -p all -m state --state ESTABLISHED,RELATED -m owner --uid-owner 108 -j RESTOREMARK
iptables -t mangle -A OUTPUT -p all -m state --state ESTABLISHED,RELATED -m owner --uid-owner 1004 -j RESTOREMARK
The server runs tinyproxy (HTTP Proxy), dante (Socks5) and rtorrent - the UIDs you see there belong to them. Now all of them work well through the VPN but rtorrent leaks packets over the venet0 interface now and then. An example for an ubuntu torrent, tcpdump output listening on venet0 (note the source ip which doesn't match the interface):
Code:
07:03:52.579107 IP 1.2.124.147.50026 > 128.143.12.130.53952: R 0:0(0) ack 1513249982 win 0
07:03:56.586871 IP 1.2.124.147.50026 > 128.143.12.130.53952: R 0:0(0) ack 1 win 0
07:04:04.602877 IP 1.2.124.147.50026 > 128.143.12.130.53952: R 0:0(0) ack 1 win 0
07:04:10.462347 IP 1.2.124.147.50020 > 71.57.111.226.44070: R 0:0(0) ack 4195453463 win 0
07:04:12.846224 IP 1.2.124.147 > 202.180.117.81: ICMP 1.2.124.147 udp port 50010 unreachable, length 101
07:04:13.459161 IP 1.2.124.147.50020 > 71.57.111.226.44070: R 0:0(0) ack 1 win 0
07:04:18.096459 IP 1.2.124.147 > 202.180.117.81: ICMP 1.2.124.147 udp port 50010 unreachable, length 101
07:04:19.493962 IP 1.2.124.147.50020 > 71.57.111.226.44070: R 0:0(0) ack 1 win 0
Same time on the tun0 iface:
Code:
07:03:50.574631 IP 128.143.12.130.53952 > 1.2.124.147.50026: S 1513249981:1513249981(0) win 65535 <mss 1368,sackOK,eol>
07:03:52.579065 IP 128.143.12.130.53952 > 1.2.124.147.50026: S 1513249981:1513249981(0) win 65535 <mss 1368,sackOK,eol>
07:03:56.586832 IP 128.143.12.130.53952 > 1.2.124.147.50026: S 1513249981:1513249981(0) win 65535 <mss 1368,sackOK,eol>
07:04:04.602836 IP 128.143.12.130.53952 > 1.2.124.147.50026: S 1513249981:1513249981(0) win 65535 <mss 1368,sackOK,eol>
07:04:10.462312 IP 71.57.111.226.44070 > 1.2.124.147.50020: S 4195453462:4195453462(0) win 65535 <mss 1368,nop,wscale 6,nop,nop,sackOK>
07:04:12.846190 IP 202.180.117.81.50010 > 1.2.124.147.50010: UDP, length 65
07:04:13.459120 IP 71.57.111.226.44070 > 1.2.124.147.50020: S 4195453462:4195453462(0) win 65535 <mss 1368,nop,wscale 6,nop,nop,sackOK>
07:04:18.096427 IP 202.180.117.81.50010 > 1.2.124.147.50010: UDP, length 65
07:04:19.493921 IP 71.57.111.226.44070 > 1.2.124.147.50020: S 4195453462:4195453462(0) win 65535 <mss 1368,nop,wscale 6,nop,nop,sackOK>
07:04:20.632631 IP 128.143.12.130.53952 > 1.2.124.147.50026: S 1513249981:1513249981(0) win 65535 <mss 1368,sackOK,eol>
Beside this, the image is downloaded flawlessly. When I quit rtorrent, there is much much more information leaked over venet0. I think is because the process doesn't exist anymore but the connections do and there isn't any UID to match against then.
I tried to prevent both by adding "iptables -A OUTPUT -o venet0 -s 1.0.0.0/8 ! -d 1.0.0.0/8 -j DROP" but then NOTHING works anymore. Anything gets blocked (?). Why is it sending over the venet0 interface in the first place? Do the packets not fit into the iptables I specified and do not get marked at all? If so, why do they have the tun0 source ip?
I am confused, I hope you can help me with these problems.
I got it to work by adding:
iptables -t mangle -A OUTPUT -m iprange --src-range "1.0.0.0"-"1.255.255.255" -j TUNMARK
But I don't know exactly why this is necessary, there is already an entry for routing this range "1.0.0.0/8 via 1.2.124.1 dev tun0". Maybe this is because of the SRC-IP rewriting and the routing decision was already made before that change?
