route, but only for a specific port and target

toredo · 03-14-2011, 05:09 AM

Hello,

I have to route some packages over the right interface.
I default route everything for the target-network over one network-interface. That works perfectly.

But i have to route packages for one specific host and one specific port over another network-interface. I tried many things with the route-command, but i think there's no possibility to route only one port?

May i can do this with iptables? I only found ways to forward some packages, which are coming in over one interface. But in my case all packages go out over one interface.

Best regards
toredo

Berhanie · 03-14-2011, 10:11 PM

i think you'll have to use "ip rule" for this. i'm not sure whether you can specify a port (read the man page), so you might have to mark the packets using iptables and then select based on the mark (ip rule ... fwmark ...). see this.

toredo · 03-15-2011, 04:25 AM

Ok, i've read much sites. And i tried much. But nothing worked correctly.

May i think bullshit, and i go the wrong way.

The problem again, but more specific:

There's a VPN-Server (i'll call it 1.2.3.4). I can connect me to it over the port 1196.
After this i have to interfaces:
eth0
tun0
Tun0 so uses a socket, which runs on eth0 and connect to 1.2.3.4:1196.
In the VPN internally i can connect to 1.2.3.4 too, but here i have some other open ports. Much more, and i need them. But by default linux routes all packages to 1.2.3.4 over eth0. So i added a route to 1.2.3.4 over tun0 with the "route add" command. The communication works now.

A few minutes later nothing works. I think the problem is: linux tries to send the packages of the openvpn-socket over tun0, because i added this rule... but of course, that's stupid and not possible...

I can't change something on the VPN-Server, everything is given. I only can change the client-configuration.

best regards
toredo

scorp1o · 03-15-2011, 05:02 AM

I succeeded with a similar scenario with the following.

I wanted some local services on my box, also running VPN, to forward some specific ports out on a specific interface (route), instead of routing it over VPN.

In my main routing table, the VPN connection (tap0) is the default route and I want some ports to go out on eth0 interface (attached to my router), it's achieved with the following:

Add an additional routing table to rt_tables file, found in

Code:

/etc/iproute2/rt_tables

content of that file:

Code:

255     local
254     main
253     default
0       unspec
85      special

85 special, is my additional routing table

Add routes to that routing table e.g.:
(Here 192.168.1.1 is my router IP and also my GW for eth0 and corresponding net is 192.168.1.0/24)

Code:

ip route add 192.168.1.0/24 dev eth0 table 85
ip route add default via 192.168.1.1 dev eth0 table 85

Then we need the following also (fwmarked packages need to look in specific routing table):

Code:

ip rule add fwmark 0x55 table 85

Which means that all packages marked with fwmark 0x55 will look in our special table (85)

But we haven't yet marked any packages so here is the specific iptables rule that marks packages:

Code:

iptables -t mangle -A OUTPUT -p tcp -m multiport --sport <local_ports> -j MARK --set-mark 0x55

Where <local_port> is the ports you want to go out on specific route, can be: 22,80 (or a range: 1024:1030 or both 22,80,1024:1030)

toredo · 03-15-2011, 08:31 AM

Ok, thanks. The post sounds well.

In my network the default gateway is 192.168.1.1 too.

Here my "/etc/iproute2/rt_tables"-file:

Code:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
85      special

Then i executed:

Code:

ip route add 192.168.1.0/24 dev eth0 table 85
ip route add default via 192.168.1.1 dev eth0 table 85
ip rule add fwmark 0x55 table 85
iptables -F
iptables -X
iptables -t mangle -A OUTPUT -p tcp -m multiport --sport 1196 -j MARK --set-mark 0x55

After this i started openvpn. Then the route to 1.2.3.4:* was created by openvpn (openvpn starts a script which executes "route add...").

But if i start "telnet 1.2.3.4:1196" and look at "tcpdump -i tun0 port 1196" the connection goes over tun0. And a few minutes later the openvpn-connection breaks like before.

I tried a few other combination with the commands. Nothing worked. Is there a way to analyse this more? I think the way which you described should work for me, but i think i made something wrong?!?

Thanks,
best regards toredo

scorp1o · 03-15-2011, 10:12 AM

Your default route is setup to be tun0 then?

When you run route -n
you see something similar to

Code:

0.0.0.0 <your_vpn_gw>          UG    0      0        0 tun0

What service is on that port anyway?
Longshot, but try to add the corresponding udp rule as well (same iptables command but put -p udp instead och -p tcp, but longshot)

Can you see if packages are marked?

Code:

iptables -t mangle -L -v

Look at the counters when you tried telnet to that port.

toredo · 03-15-2011, 04:55 PM

I add the route with this command:

Code:

route add -net 1.2.3.0 netmask 255.255.255.0 gw 10

May i have to use later some other servers over VPN which starts with 1.2.3.X so i add the 24bit iprange for 1.2.3.X. These are all public ip's too.

After this "route -n" shows this:

Code:

1.2.3.0      10.255.0.5      255.255.255.0   UG    0      0        0 tun0

here my fully executed commands (i replaced the network-id with 1.2.3.0):

Code:

route add -net 1.2.3.0 netmask 255.255.255.0 gw 10.255.0.5
ip route add 192.168.1.0/24 dev eth0 table 85
ip route add default via 192.168.1.1 dev eth0 table 85
ip rule add fwmark 0x55 table 85
iptables -F
iptables -X
iptables -t mangle -A OUTPUT -p tcp -m multiport --sport 1196 -j MARK --set-mark 0x55
iptables -t mangle -A OUTPUT -p udp -m multiport --sport 1196 -j MARK --set-mark 0x55

then i logged tun0 port 1196 with tcpdump (i replaced the ip with 1.2.3.4):

Code:

umcgw:~# tcpdump -i tun0 port 1196
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
22:48:13.805157 IP 192.168.1.22.52337 > 1.2.3.4.1196: P 2749728642:2749728745(103) ack 1117251021 win 6672 <nop,nop,timestamp 2080 2705890242>
22:48:29.684980 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 3668 2705923109>
22:48:29.875237 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 3687 2705923165,nop,nop,sack 1 {1:56}>
22:48:30.323375 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 3731 2705923277,nop,nop,sack 1 {1:56}>
22:48:31.218944 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 3821 2705923501,nop,nop,sack 1 {1:56}>
22:48:33.011465 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 4000 2705923949,nop,nop,sack 1 {1:56}>
22:48:36.594792 IP 192.168.1.22.52337 > 1.2.3.4.1196: . ack 56 win 6672 <nop,nop,timestamp 4358 2705924845,nop,nop,sack 1 {1:56}>
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel

So the packages to 1.2.3.4:1196 where not went over eth0. If i log eth0 with the port 1196 nothing happens.

The output of "iptables -t mangle -L -v":

Code:

...
Chain OUTPUT (policy ACCEPT 805 packets, 107K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       tcp  --  any    any     anywhere             anywhere            multiport sports 1196 MARK xset 0x55/0xffffffff
    0     0 MARK       udp  --  any    any     anywhere             anywhere            multiport sports 1196 MARK xset 0x55/0xffffffff
...

The packages where count up. So this step should work.

best regards & thanks
toredo

scorp1o · 03-16-2011, 02:03 AM

Long shot again..

Try do a
ip route flush cache after you've added the rules i.e.

Code:

route add -net 1.2.3.0 netmask 255.255.255.0 gw 10.255.0.5
ip route add 192.168.1.0/24 dev eth0 table 85
ip route add default via 192.168.1.1 dev eth0 table 85
ip rule add fwmark 0x55 table 85
ip route flush cache
iptables -F
iptables -X
iptables -t mangle -A OUTPUT -p tcp -m multiport --sport 1196 -j MARK --set-mark 0x55
iptables -t mangle -A OUTPUT -p udp -m multiport --sport 1196 -j MARK --set-mark 0x55

You saw the zeros out to left of the output:

Code:

iptables -t mangle -L -v

Got incremented i.e. not zero after you ran:
telnet 1.2.3.4:1196 ?

Also when you run

Code:

ip route show table special

You see you specific routing table? i.e. something like
192.168.1.0/24 dev eth0 table 85
default via 192.168.1.1 dev eth0 table 85

Starting to run out of ideas I'm afraid =/

toredo · 03-16-2011, 03:07 AM

Ok. I'll try it again. But i will document every step now.

First i boot the device. Then the VPN-Daemon starts and creates a new route.
Output of route -n:

Code:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.255.0.5      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.255.0.1      10.255.0.5      255.255.255.255 UGH   0      0        0 tun0
1.2.3.0      10.255.0.5      255.255.255.0   UG    0      0        0 tun0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0

Which interfaces are available?

Code:

umcgw:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:c2:8b:b4:ce
          inet addr:192.168.1.22  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::250:c2ff:fe8b:b4ce/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:487 errors:1 dropped:0 overruns:0 frame:0
          TX packets:377 errors:1 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:1000
          RX bytes:51509 (50.3 KiB)  TX bytes:56265 (54.9 KiB)
          Interrupt:21 Base address:0x4000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2549 (2.4 KiB)  TX bytes:2549 (2.4 KiB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.255.0.6  P-t-P:10.255.0.5  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:748 (748.0 B)  TX bytes:8492 (8.2 KiB)

Ok, fine i check /etc/iproute/rt_tables...:

Code:

umcgw:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
85      special

I think that's ok too. So i execute this code:

Code:

umcgw:~# ip route add default via 192.168.1.1 dev eth0 table 85
umcgw:~# ip rule add fwmark 0x55 table 85
umcgw:~# ip route flush cache
umcgw:~# iptables -F
umcgw:~# iptables -X
umcgw:~# iptables -t mangle -A OUTPUT -p tcp -m multiport --sport 1196 -j MARK --set-mark 0x55
umcgw:~# iptables -t mangle -A OUTPUT -p udp -m multiport --sport 1196 -j MARK --set-mark 0x55
umcgw:~# ip route flush cache
umcgw:~#

Now i test the following:

Code:

umcgw:~# telnet 1.2.3.4 1196
Trying 1.2.3.4...

On the same time i run tcpdump:

Code:

umcgw:~# tcpdump -i tun0 port 1196
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
08:58:44.526632 IP 10.255.0.6.47182 > 1.2.3.4.1196: S 3370931735:3370931735(0) win 5840 <mss 1460,sackOK,timestamp 19061 0,nop,wscale 1>
08:58:47.520000 IP 10.255.0.6.47182 > 1.2.3.4.1196: S 3370931735:3370931735(0) win 5840 <mss 1460,sackOK,timestamp 19361 0,nop,wscale 1>

Hm. Ok i try "iptables -t mangle -L -v":

Code:

umcgw:~# iptables -t mangle -L -v
Chain PREROUTING (policy ACCEPT 313 packets, 29119 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 313 packets, 29119 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 315 packets, 31629 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       tcp  --  any    any     anywhere             anywhere            multiport sports 1196 MARK xset 0x55/0xffffffff
    0     0 MARK       udp  --  any    any     anywhere             anywhere            multiport sports 1196 MARK xset 0x55/0xffffffff

Chain POSTROUTING (policy ACCEPT 315 packets, 31629 bytes)
 pkts bytes target     prot opt in     out     source               destination

Hm, no packages were marked. No changes if i try some telnet-connections again. In the last post i looked to the worse value...

At least:

Code:

umcgw:~# ip route show table special
192.168.1.0/24 dev eth0  scope link
default via 192.168.1.1 dev eth0

Im confused. The problem sounds so easy, but the solution is so difficult.

thanks & best regards
toredo

-edit-
Forgot to call "ip route flush cache". I'll restart the device, test it and edit my post.

scorp1o · 03-16-2011, 06:21 AM

I think I see the issue here, I assumed you ran a local service, at that port 1196 (that is my scenario namely, but not what you want to achieve)

Try to change the the iptables rules to the following:

Code:

iptables -t mangle -A OUTPUT -p tcp -m multiport --dport 1196 -j MARK --set-mark 0x55
iptables -t mangle -A OUTPUT -p udp -m multiport --dport 1196 -j MARK --set-mark 0x55

And come back with what happens after

Code:

telnet 1.2.3.4:1196

and

Code:

iptables -t mangle -L -v

toredo · 03-16-2011, 08:05 AM

The new output is:

Code:

umcgw:~# iptables -t mangle -L -v
Chain PREROUTING (policy ACCEPT 1514 packets, 111K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1514 packets, 111K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1376 packets, 166K bytes)
 pkts bytes target     prot opt in     out     source               destination
  520 82919 MARK       tcp  --  any    any     anywhere             anywhere            multiport dports 1196 MARK xset 0x55/0xffffffff
    0     0 MARK       udp  --  any    any     anywhere             anywhere            multiport dports 1196 MARK xset 0x55/0xffffffff

Chain POSTROUTING (policy ACCEPT 1376 packets, 166K bytes)
 pkts bytes target     prot opt in     out     source               destination

Now it works perfectly! Big thanks for your help

best regards
toredo

scorp1o · 03-16-2011, 12:27 PM

Great!

Glad I could help, since I had a similar issue myself and figured out the solution on my own, I'll try to help as many as possible with these kind of issues

/scorp1o

daviest11 · 07-12-2011, 03:14 AM

Hopefully I can save someone else the weeks of pain I had, with this not working for no apparent reason!

From [URL="http://blog.wpkg.org/2010/09/13/source-port-routing/"]

Quote:

Still doesn’t work? Check these things below:

rp_filter has to be set to 0 for given interfaces – 0 is the default value set by the Linux kernel, but some distributions (i.e. Ubuntu, Mandriva) alter it and set it to 1; just adding that to /etc/sysctl.conf should do the trick to make sure this value is set to 0 after reboot:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

If you’re not rebooting any time soon, see what these value currently are with:

find /proc/sys/net -name rp_filter | xargs cat

After I'd made that change, it all worked beautifully!