[SOLVED] Policy routing using two point-to-point links

Dunkelschorsch · 08-27-2009, 07:47 AM

Hi Folks!

This one has been driving me nuts for some days now:

My Gentoo box which is acting as an internet gateway has two point-to-point interfaces, ppp0 (PPPoE to my ISP) and ppp1 (PPTP VPN link to IPREDator). Packets from my local network are just routed through ppp0 and now the fun part starts: I want to MARK (netfilter...) all packets originating from one specific user on that box in order to use another routing table that will contain a default route via the ppp1 interface.

Marking seems to work fine as does the second routing table. But quite mysteriously (at least for me), the packets sent out on ppp1 contain the wrong source IP address, namely the address associated with ppp0.

So here is what ifconfig and friends tell me:

Network interfaces:

Code:

# ifconfig ppp0
ppp0      Protokoll:Punkt-zu-Punkt Verbindung
          inet Adresse:87.175.89.246  P-z-P:217.0.117.7  Maske:255.255.255.255
          UP PUNKTZUPUNKT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:126202 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88118 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:3
          RX bytes:65312881 (62.2 MiB)  TX bytes:9591159 (9.1 MiB)

Code:

# ifconfig ppp1
ppp1      Protokoll:Punkt-zu-Punkt Verbindung
          inet Adresse:93.182.152.74  P-z-P:93.182.152.2  Maske:255.255.255.255
          UP PUNKTZUPUNKT RUNNING NOARP MULTICAST  MTU:1496  Metric:1
          RX packets:338 errors:0 dropped:0 overruns:0 frame:0
          TX packets:332 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:3
          RX bytes:29658 (28.9 KiB)  TX bytes:33840 (33.0 KiB)

Routing tables:

Code:

# ip route list table main
217.0.43.161 dev ppp0  scope link
217.0.43.177 dev ppp0  scope link
93.182.152.2 dev ppp0  scope link
217.0.117.7 dev ppp0  proto kernel  scope link  src 87.175.89.246
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1
127.0.0.0/8 dev lo  scope link
default dev ppp0  scope link

Code:

# ip route list table IPREDator
217.0.43.161 dev ppp0  scope link
217.0.43.177 dev ppp0  scope link
93.182.152.2 dev ppp0  scope link
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1
127.0.0.0/8 dev lo  scope link
default dev ppp1  scope link

Routing policy:

Code:

# ip rule list
0:      from all lookup local
10000:  from all fwmark 0x4 lookup IPREDator
32766:  from all lookup main
32767:  from all lookup default

The source of this problem is obviously the mismatch between the ppp1 interface and its IP address. Now look at this:

ping as root using ppp1:

Code:

# ping -I ppp1 www.kernel.org
PING pub.eu.kernel.org (130.239.17.4) from 93.182.151.30 ppp1: 56(84) bytes of data.
64 bytes from pub4.kernel.org (130.239.17.4): icmp_seq=1 ttl=56 time=139 ms

"from 93.182.151.30 ppp1" is exactly what I want to read.

ping as a non-root user. And not the packet mangling special one, either:

Code:

# ping -I ppp1 www.kernel.org
PING pub.eu.kernel.org (130.239.17.4) from 87.175.89.246 ppp1: 56(84) bytes of data.
64 bytes from pub4.kernel.org (130.239.17.4): icmp_seq=1 ttl=56 time=361 ms

"from 87.175.89.246 ppp1" ?!?! How is that possible? This is the address for ppp0. An there is no routing, mangling, filtering, whatever involved at this point. I am totally at a loss here.

Your help would be very much appreciated!

Edit: If I use ppp1 for the default route everything works as expected.

kbp · 08-27-2009, 09:17 AM

Hi Dunkelschorsch,

I just wondered if you could run tcpdump to confirm which interface the packet is actually exiting.. ?

cheers,

kbp

Dunkelschorsch · 08-27-2009, 09:38 AM

Of course. I knew, I still forgot something. Here it is:

Code:

# tcpdump -n -i ppp1 dst host www.kernel.org
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp1, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
16:26:15.658717 IP 93.182.151.30 > 130.239.17.4: ICMP echo request, id 26719, seq 1, length 64
16:26:48.966065 IP 93.182.151.30 > 130.239.17.4: ICMP echo request, id 27487, seq 1, length 64

The first packet was generated by the non-root user, the second by root. So the interface AND the address are correct here, despite what ping claimed. I do not know if that's good though...

And here comes the tcpdump output when the special user tries to establish an ssh-connection to 131.188.138.163.

Code:

# tcpdump -n -i ppp1 dst port 22 and dst host 131.188.138.163
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp1, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
16:35:22.164081 IP 87.175.89.246.53821 > 131.188.138.163.22: S 1875886089:1875886089(0) win 5808 <mss 1452,sackOK,timestamp 18619725 0,nop,wscale 5>
16:35:25.162629 IP 87.175.89.246.53821 > 131.188.138.163.22: S 1875886089:1875886089(0) win 5808 <mss 1452,sackOK,timestamp 18620475 0,nop,wscale 5>
16:35:31.162633 IP 87.175.89.246.53821 > 131.188.138.163.22: S 1875886089:1875886089(0) win 5808 <mss 1452,sackOK,timestamp 18621975 0,nop,wscale 5>

The correct interface is chosen according to the "IPREDator" routing table, but the source address is, this time for real, 87.175.89.246, ppp0's address.

evilted · 08-27-2009, 10:08 AM

yes. it is correct. theres a sysctl option to disable this, and allow the origin of the packet to take its true source, however i forget what it was, sorry, but at least you got somewhere to start looking!

kbp · 08-27-2009, 10:12 AM

So the problem ( async route? )occurs for normal users and the 'special' user ? .. then it wouln't be related to the IPREDator routing mods ?

<edit>
Ah.. evilted, were you thinking of: net.ipv4.conf.all.accept_source_route ?
</edit>

kbp

Dunkelschorsch · 08-27-2009, 10:23 AM

Normal users are not affected. Their packets use the ppp0 interface as specified in the "main" routing table. Only the special one's packets, which are marked "0x4", use the "IPREDator" table and use, more or less correctly, ppp1. But their source IP, unfortunately does not reflect that.

I will take a look at that accept_source_route thingy. *keeping fingers crossed*

Dunkelschorsch · 08-27-2009, 10:39 AM

No, that didn't do anything...

I played around with some of the other variables as well, to no avail.

Oops, one important thing I have not mentioned yet: access from the outside (only tried ssh), via the ipredator address works flawlessly.

kbp · 08-27-2009, 11:56 PM

I think we may need a linux networking guru for this one, I'm out of inspiration ... anybody?

Dunkelschorsch · 08-28-2009, 03:47 AM

So this:

Code:

# iptables -t nat -A POSTROUTING -o ppp1 -m owner --uid-owner 1000 -j SNAT --to 93.182.151.51

did the trick!

But I still wonder why that is necessary at all

<edit>
maybe this is slightly better since the packets to be routed are already marked:

Code:

iptables -t nat -A POSTROUTING -o ppp1 -m mark --mark 0x4 -j SNAT --to 93.182.151.51

</edit>

TimothyEBaldwin · 08-30-2009, 02:26 PM

Source address selection does not use iptables as no packets are involved. Network namespaces may be useful.