Send traffic from one application out a specific interface?

xamindar · 03-02-2010, 10:29 PM

I have a server set up with various services and it has two different internet links. One is through eth0 and the other is through eth1.
eth0 is the default route.

I am trying to set up deluge to use ONLY the eth1 interface for all it's traffic but I can't seem to get it to work.
I tried following the howto here: http://linux-ip.net/html/adv-multi-internet.html
and just modifying it to my application but it isn't working.

Here is the firewall script I have set up on this machine. Trying to keep it simple right now and will expand it as I set everything up.
Again, I am trying to have everything go out eth0 EXCEPT deluge traffic in which I want to go out(and come back in) eth1. But I must be missing something.

Code:

#!/bin/bash

## Variables applying to the system
IPTABLES='/sbin/iptables'
### Modules needed, just add one per line.
MODULES="ip_tables 
	 iptable_nat
	 ip_nat_ftp
	 ip_conntrack_ftp"
for i in $MODULES;
do 
 echo "Inserting module $i"
 modprobe $i
done

# Flush rules and delete chains
$IPTABLES -F
$IPTABLES -X
$IPTABLES -F -t nat
$IPTABLES -F -t mangle
# Set the default policies for the chains
$IPTABLES -t filter -P INPUT DROP
$IPTABLES -t filter -P OUTPUT ACCEPT
$IPTABLES -t filter -P FORWARD DROP
$IPTABLES -t nat -P PREROUTING ACCEPT
$IPTABLES -t nat -P POSTROUTING ACCEPT
$IPTABLES -t nat -P OUTPUT ACCEPT

### Set up the firewall rules
# Allow all connections established by me (because default is to drop)
$IPTABLES -t filter -A INPUT -i lo -j ACCEPT
# Allow anything from trusted lan1 to this box
$IPTABLES -t filter -A INPUT -i eth0 -j ACCEPT
# Allow anything from outside of lan2 in if connection is already established 
$IPTABLES -t filter -A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT

######################################################################
#########The following is for routing to specific interfaces##########
######################################################################
DTABLE=3
STABLE=main
MARK=3

#Create a second routing table with a new default gateway for the second interface (eth1)
ip route flush table $DTABLE
ip route show table $STABLE | grep -Ev '^default' \
  | while read ROUTE ; do
    ip route add table $DTABLE $ROUTE
done

IP1=192.168.1.253 #IP on eth1
P1=192.168.1.254 #Gateway on eth1

ip route add default via $P1 table $DTABLE
ip rule add from $IP1 table $DTABLE
ip rule add fwmark $MARK table $DTABLE


#Mark deluged torrent packets so they will be used by the new routing table
$IPTABLES -t mangle -A OUTPUT -m owner --uid-owner deluge -j MARK --set-mark $MARK

$IPTABLES -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.253


#######################################################################
###############END INTERFACE ROUTING###################################
#######################################################################

I have it create a second routing table and then the iptables rules are supposed to mark it to go out using that routing table. But I must have something wrong here. It seems they go out but can't get back in.

Code:

~ # ip route show table 3
172.16.0.0/24 dev eth0  proto kernel  scope link  src 172.16.0.3  metric 2
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.253  metric 2005
127.0.0.0/8 via 127.0.0.1 dev lo
default via 192.168.1.254 dev eth1
~ # ip route show table main
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.253  metric 2005
172.16.0.0/24 dev eth0  proto kernel  scope link  src 172.16.0.3  metric 2
127.0.0.0/8 via 127.0.0.1 dev lo
default via 172.16.0.1 dev eth0  metric 2
default via 192.168.1.254 dev eth1  metric 2005

Code:

~ # ip rule show
0:      from all lookup local
32764:  from all fwmark 0x3 lookup 3
32765:  from 192.168.1.70 lookup 3
32766:  from all lookup main
32767:  from all lookup default

With all this I'm not quite sure how it all fits together. Could someone help me out here? Thanks in advance.

EDIT: corrected eth1 ip address to current one.

leslieviljoen · 03-03-2010, 03:58 AM

One thing to remember is that Deluge is going to be accepting NEW incoming connections. You need to make sure those new incoming connections are allowed, and are coming in on the right interface.

When you say it is not working, can you be more specific? You should be able to use tcpdump to confirm which interface things are happening on. Try watching a normal conversation with a specific peer over the default gateway and no firewall, then set up your firewall and watch another conversation with a peer to understand what is happening.

xamindar · 03-03-2010, 06:21 AM

Quote:

Originally Posted by leslieviljoen

One thing to remember is that Deluge is going to be accepting NEW incoming connections. You need to make sure those new incoming connections are allowed, and are coming in on the right interface.

I tried removing the default drop clauses in the firewall so there was nothing blocking anything from receiving new connections and it didn't change a thing. Something with the marking is preventing it from working correctly or I am missing something. I can't find any info online on how to really do this so maybe it is impossible.

Code:

$IPTABLES -t mangle -A OUTPUT -m owner --uid-owner deluge -j MARK --set-mark $MARK

$IPTABLES -t nat -A POSTROUTING -o $EVILIF -j SNAT --to-source $IP1

When those lines are removed deluge is able to work again (with the default route on eth0 which is not what I want). I can't believe how hard it is to get one program to use a specific interface. It sounded like an easy thing when I started but turned out the complete opposite.

If I run tcpdump on eth1 I get a lot of this:

Code:

04:16:41.381988 IP 79.103.172.93.22116 > 192.168.1.253.65123: UDP, length 56
04:16:41.415829 IP 81.102.111.72.16005 > 192.168.1.253.candp: Flags [S.], seq 1126976333, ack 4063673460, win 5792, options [mss 1418,sackOK,TS val 3085181774 ecr 2879983,nop,wscale 6], length 0
04:16:41.489227 IP 172.16.0.3.65123 > 58.12.63.253.25359: Flags [S.], seq 4094652537, ack 2584070184, win 5840, options [mss 1460,nop,nop,sackOK], length 0
04:16:41.510773 IP 192.168.1.253.46243 > 62.31.148.26.17773: Flags [S], seq 4243710982, win 5840, options [mss 1460,sackOK,TS val 2881017 ecr 0,nop,wscale 6], length 0
04:16:41.520500 IP 192.168.1.253.45387 > 95.160.214.219.17177: Flags [S], seq 4194907818, win 5840, options [mss 1460,sackOK,TS val 2881018 ecr 0,nop,wscale 6], length 0
04:16:41.553812 ARP, Request who-has 192.168.1.254 (Broadcast) tell 192.168.1.71, length 46
04:16:41.570963 IP 192.168.1.253.35494 > 88.195.83.220.47001: Flags [S], seq 4251066667, win 5840, options [mss 1460,sackOK,TS val 2881023 ecr 0,nop,wscale 6], length 0

But I'm not sure what that means. Looks like it's communicating. In the deluge ui it shows a low amount of connections (around 90 out of a possible 400)which constantly fluctuates. It never receives or sends any actual data though. So it seems deluge is trying to establish connections but fails or they time out.

nimnull22 · 03-03-2010, 10:18 AM

Quote:

Originally Posted by xamindar

...
I am trying to set up deluge to use ONLY the eth1 interface for all it's traffic but I can't seem to get it to work.
...

EDIT: corrected eth1 ip address to current one.

Can you tell, please, which direction traffic will be expected. Outgoing only, or incoming as well?

And another question: does a user "deluge" exist?

Thanks

xamindar · 03-03-2010, 06:34 PM

Quote:

Originally Posted by nimnull22

Can you tell, please, which direction traffic will be expected. Outgoing only, or incoming as well?

And another question: does a user "deluge" exist?

Thanks

Yes outgoing and incoming. And yes the deluge user exists (hence the "-m owner --uid-owner deluge"). The deluged daemon is run under that user. But if there is a better way to mark packets I'm all ears. It doesn't seem to be working this way.

ericje · 09-11-2010, 02:10 PM

Quote:

Originally Posted by xamindar

Yes outgoing and incoming. And yes the deluge user exists (hence the "-m owner --uid-owner deluge"). The deluged daemon is run under that user. But if there is a better way to mark packets I'm all ears. It doesn't seem to be working this way.

(this reply is a little late, but I found this post with google, and other people might find the following useful too)

Disabling the rp_filter should make it work, I don't see any other issues with your setup. Note, you have to do both 'all' and 'eth1' if you're using 2.6.31+, since they changed the meaning a little bit.

Code:

echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter
echo 0 >/proc/sys/net/ipv4/conf/eth1/rp_filter

dyinman · 08-08-2011, 06:22 PM

Sorry this reply is late, but there's so little current reading out there on how to pull something like this off.

Did you ever get it working?

I have almost the exact same setup, although a little simpler, and I am unable to get it working either.