IP Routing policies for separating traffic based on ports

BhushanPathak · 08-20-2014, 09:12 AM

Hello,

I am running CentOS 5 with kernel version kernel-2.6.18-371.8.1.el5

I am referring the online how-to guide for advanced linux routing -
http://www.tldp.org/HOWTO/Adv-Routin...netfilter.html

The requirement that I have is that based on the port being used for communication, the traffic should flow through different network interfaces, for ex -
eth0 should be used -
1. Accept & respond to requests coming over port 80, 443, 8080
2. When the server wants to start communication outside on these ports

eth1 should be used -
1. Accept & respond to requests coming over port 1099, 1399
2. When the server wants to start communication outside on these ports

To do that, I wrote the following script -

Code:

#!/bin/bash

. /etc/profile

## Network Interface 1 details - eth0
IF1=eth0             #Name
IP1=1.1.1.61    #IP Address
P1=1.1.1.1      #Gateway
P1_NET=255.255.255.0 #Netmask

## Network Interface 2 details - eth1
IF2=eth1             #Name
IP2=1.1.1.67    #IP Address
P2=1.1.1.1      #Gateway
P2_NET=255.255.255.0 #Netmask

## Define routing table for each interface
echo 201 T1 >> /etc/iproute2/rt_tables #Routing table for eth0
echo 202 T2 >> /etc/iproute2/rt_tables #Routing table for eth1

## To route answers to packets coming in over a particular interface, say eth0, back out again over that same interface

# 1. Build a route to the gateway and build a default route via that gateway
ip route add $P1_NET dev $IF1 src $IP1 table T1
ip route add default via $P1 table T1
ip route add $P2_NET dev $IF2 src $IP2 table T2
ip route add default via $P2 table T2

# 2. It is a good idea to route things to the direct neighbour through the interface connected to that neighbour
ip route add $P1_NET dev $IF1 src $IP1
ip route add $P2_NET dev $IF2 src $IP2

# 3. Preference for default route
ip route add default via $P2

# 4. Set up the routing rules. These actually choose what routing table to route with
ip rule add from $IP1 table T1
ip rule add from $IP2 table T2

# 5. Flush the route cache
ip route flush cache

####### TEST ROUTING POLICY FOR PORT 22 OUTGOING CONNECTION ###########
iptables -A PREROUTING -t mangle -p tcp --dport 22 -j MARK --set-mark 1
ip rule add fwmark 1 table T2

ip route flush cache

I added the "TEST ROUTING POLICY FOR PORT 22 OUTGOING CONNECTION" section in the script. The section was written with the following intention -
If I open a SSH session from the OS to another linux machine [1.1.1.70], interface eth1 [1.1.1.67] should be used. I executed the script & started the SSH session. The SSH session was established, but to verify that eth1 interface was used, I executed netstat command on 1.1.1.70, which gave the following output -

Code:

[root@OS4 tmp]# netstat -anpt | grep sshd
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1773/sshd
tcp        0     52 1.1.1.70:22            1.1.1.231:2292                               ESTABLISHED 32271/sshd
tcp        0      0 1.1.1.70:22            1.1.1.61:60768                               ESTABLISHED 32371/sshd
tcp        0      0 1.1.1.70:22            1.1.1.61:51396                               ESTABLISHED 32513/sshd
tcp        0      0 :::22                       :::*                        LISTEN      1773/sshd

I expected the 1.1.1.67 IP address to turn up in the above output, instead I get eth0 IP - 1.1.1.61.

The 1.1.1.231 is my Linux desktop IP address, from which I access both the machines [1.1.1.61 & 1.1.1.67].

Anything wrong with my script? Needs updates? More rules to be defined? OR I got the understanding the wrong way?

Thanks
- Bhushan Pathak

BhushanPathak · 08-20-2014, 10:01 AM

The sysadmin has executed the following on the OS [1.1.1.61/67], where I am trying to define the routing policies -

Code:

echo "net.ipv4.tcp_timestamps = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.default.send_redirects = 0" >> /etc/sysctl.conf
echo "net.ipf4.conf.all.send_redirects = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.accept_redirects = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.accept_source_route = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.secure_redirects = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.log_martians = 1" >> /etc/sysctl.conf
echo "net.ipv4.conf.default.accept_redirects = 0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.rp_filter = 1" >> /etc/sysctl.conf

Would these affect?

- Bhushan

cellarweasel · 08-21-2014, 07:39 PM

Hey Bhushan,
Couple of questions for you;
Is this a machine that you have a root account on?
Is this a machine that you would be able to recompile the kernel on?

The reason I ask those two questions is because the kernel includes that are required at the bottom I don't think are included by default in any distro that would normally be used as a server/desktop. I would use this post to check for certain:
http://unix.stackexchange.com/questi...option-enabled

I also have a meta question: Why do you need to do this in this way? I would use a firewall and create Outgoing rules. This would only allow my computer when sending packets out of certain interfaces. In this solution nothing would interact with routing.

Hopefully this gives you some things that will move you forward. Let me know if anything didn't make sense.

Evan

BhushanPathak · 08-22-2014, 01:29 AM

Hello Evan,

Yes, I have a root account on the machine & can also look into recompiling the kernel if needed [this would a first time for me]. I will check the kernel options, but if not enabled, do you know how to get the kernel compiled with the required options?

I looked into the IP routing policies because of a previous post in LQ which pointed me in that direction -
http://www.linuxquestions.org/questi...on-4175505551/

I can also checkout the firewall only option, but would need some guide/documentation to help me get through.

Thanks for your time, appreciate it. I was shooting in the dark earlier.

Thanks
Bhushan Pathak

BhushanPathak · 08-22-2014, 02:39 AM

All the kernel options are enabled in the config file -

Code:

root >cd /boot/
root >ls
config-2.6.18-371.9.1.el5  initrd-2.6.18-371.9.1.el5.img  symvers-2.6.18-371.9.1.el5.gz  vmlinuz-2.6.18-371.9.1.el5
grub                       lost+found                     System.map-2.6.18-371.9.1.el5
root >uname -r
2.6.18-371.9.1.el5
root >grep CONFIG_IP_ADVANCED_ROUTER /b
bin/  boot/ 
root >grep CONFIG_IP_ADVANCED_ROUTER /boot/config-2.6.18-371.9.1.el5 
CONFIG_IP_ADVANCED_ROUTER=y
root >grep CONFIG_IP_MULTIPLE_TABLES /boot/config-2.6.18-371.9.1.el5 
CONFIG_IP_MULTIPLE_TABLES=y
root >grep CONFIG_IP_ROUTE_FWMARK /boot/config-2.6.18-371.9.1.el5 
CONFIG_IP_ROUTE_FWMARK=y
root >

cellarweasel · 08-25-2014, 07:20 PM

Bhushan,
That is good to hear! I suppose I was wrong and those have made their way into the kernel as a default these days.

So I was looking over your script and found something that I think may be off by a something. P1_NET and P2_NET both equal 255.255.255.0 and have the comment of #Netmask however in your ip route commands they are used as the network's address for routing. I'm not sure if this is it at all so let me know.

I found this in the ip (iproute2) command reference pdf:
Examples:
1. Add a plain route to network 10.0.0.0/24 via gateway 193.233.7.65

Code:

ip route add 10.0.0/24 via 193.233.7.65

Or look at the html version here http://linux-ip.net/gl/ip-cref/node78.html
Let me know what you find.

-Evan

BhushanPathak · 09-17-2014, 01:28 AM

Hello Evan,

I have been swamped with some other priority work, hence been unable to try out your suggestions. Am trying to finish my current work ASAP & get back on this.

Thanks
Bhushan