Problem
my lan network is 192.168.0.X/255.255.255.0
I have a server that provides dhcp, dns, and is a gateway.
The server is also a file server.
The server has 2 nic's for the lan, eth0 is 192.168.0.2 and eth1 is 192.168.0.1
I want all internet traffic to go over eth1 (also dns/dhcp)
I want all other traffic to go over eth0
What is working
Currently I have both interfaces responding to requests properly, as in if I ping 192.168.0.1 it responds from .1, if I ping .2 is responds on .2
I also have it set up so that any time I try to access a system on the lan from the server it uses eth0.
Where it breaks
HOWEVER, when I try to visit a web site with only eth1 connected it will resolve the dns and send the packets, but the packets do not return.
Testing / Trial and error
The computer I am using to test this can only be connected to the servers eth0 or eth1, not both at once, so during my testing I am 100% sure if I am, getting a response it is on the proper interface.
If I use either interface alone w/ the other disabled it works fine, but that is not what I need.
If I configure the server to use eth1 by default instead of eth0 when it accesses the lan computers then the setup works fine in every way except that connections established from the server are on the internet interface instead of the general one.
I have been playing with ip route and iptables for about 16 hours now. At one point 1 hour ago I finally found 2 commands that I ran at the shell that made it work as I wanted, so I added those lines to my config script and restarted... it does not work, so an option that I did not think was important from that shell session probably was important, but I have reviewed my bash history and none of it did it :-(
The Setup
I currently have a very complicated setup. 2 ISP's each providing a seperate internet connection. My server then takes both connections and uses 'ip route' to utilize both connections.
The server then provides dns using dnsmasq, and uses iptables to handle routing.
There are 2 seperate internal networks, one is eth4 192.168.1.1 and is bandwidth limited to 128000
the other is the lan in question w/ 2 interfaces
The script that handles it all
Code:
#The 'main' lan network interface
/sbin/ifconfig eth0 192.168.0.2 netmask 255.255.255.0
/sbin/ip route del 192.168.0.0/24 dev eth0
/sbin/ip route add 192.168.0.0/24 dev eth0 src 192.168.0.2 table 1
/sbin/ip rule add from 192.168.0.2 table 1
#The 'internet' providing interface
/sbin/ifconfig eth1 192.168.0.1 netmask 255.255.255.0
/sbin/ip route del 192.168.0.0/24 dev eth1
/sbin/ip route add 192.168.0.0/24 dev eth1 src 192.168.0.1 table 2
/sbin/ip rule add from 192.168.0.1 table 2
#The other internal network that is limited
/sbin/ifconfig eth4 192.168.1.1 netmask 255.255.255.0
#ISP 1
/sbin/ifconfig eth2 71.39.157.170 netmask 255.255.255.248
/sbin/ip route add 71.39.157.168/29 dev eth2 src 71.39.157.170 table 3
/sbin/ip route add default via 71.39.157.174 table 3
/sbin/ip rule add from 71.39.157.170 table 3
#ISP2
/sbin/ifconfig eth3 192.168.100.2 netmask 255.255.255.0
/sbin/ip route add 192.168.100.0/24 dev eth3 src 192.168.100.2 table 4
/sbin/ip route add default via 192.168.100.1 table 4
/sbin/ip rule add from 192.168.100.2 table 4
#Switch between ISP's on new connections to utilize all bandwidth
/sbin/ip route add default scope global nexthop via 71.39.157.174 dev eth2 weight 1\
nexthop via 192.168.100.1 dev eth3 weight 3
#If mark 80 is present in the [packets/fames/?] use route table 2 (the internet providing interface) This is what I thought made it work when it was working for a short time.
/sbin/ip rule add fwmark 80 table 2
# Default interface to use for 192.168.0.0 should be eth0
/sbin/ip route add 192.168.0.0/24 dev eth0 src 0.0.0.0
# Try to route traffic from ISP 1 to eth1 if it is destined for 192.168.0.0... this did not work
/sbin/ip route append 192.168.0.0/24 dev eth1 src 71.39.157.170
# Try to route traffic from ISP 2 to eth1 if it is destined for 192.168.0.0... this did not work
/sbin/ip route append 192.168.0.0/24 dev eth1 src 192.168.100.2
#Set the nameserver(s)
echo "nameserver 192.168.100.1" > /etc/resolv.conf
echo "nameserver 71.39.157.174" >> /etc/resolv.conf
#Throttle traffic on the other internal network
/sbin/tc qdisc add dev eth4 root handle 1: htb default 1
/sbin/tc class add dev eth4 parent 1: classid 1:1 htb rate 128kbit burst 15k
#Set some variables for the iptables stuff.
WIFI="eth4"
LOCAL="eth1"
CABLE="eth3"
DSL="eth2"
iptables -F
iptables -t nat -F
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD DROP
#Then we lock our services so they only work from the LAN
iptables -I INPUT 1 -i ${LOCAL} -j ACCEPT
iptables -I INPUT 1 -i ${WIFI} -j ACCEPT
iptables -I INPUT 1 -i lo -j ACCEPT
iptables -A INPUT -p UDP --dport bootps -i ! ${LOCAL} -j REJECT
iptables -A INPUT -p UDP --dport bootps -i ! ${WIFI} -j REJECT
iptables -A INPUT -p UDP --dport domain -i ! ${LOCAL} -j REJECT
iptables -A INPUT -p UDP --dport domain -i ! ${WIFI} -j REJECT
#(Optional) Allow access to our server from the WAN
for i in '80' '8080' '20' '21' '22' '24' '25' '37' '110' '113' '118' '123' '156' '194' '220' '389' '443' '465' '531' '989' '990' '993' '995'; do
iptables -A INPUT -p TCP --dport ${i} -i ${CABLE} -j ACCEPT
iptables -A INPUT -p TCP --dport ${i} -i ${DSL} -j ACCEPT
done
#Drop TCP / UDP packets to privileged ports
iptables -A INPUT -p TCP -i ! ${LOCAL} -d 0/0 --dport 0:1023 -j DROP
iptables -A INPUT -p UDP -i ! ${LOCAL} -d 0/0 --dport 0:1023 -j DROP
iptables -A INPUT -p TCP -i ! ${WIFI} -d 0/0 --dport 0:1023 -j DROP
iptables -A INPUT -p UDP -i ! ${WIFI} -d 0/0 --dport 0:1023 -j DROP
#Finally we add the rules for NAT
iptables -I FORWARD -i ${WIFI} -d 192.168.1.0/255.255.255.0 -j DROP
iptables -A FORWARD -i ${WIFI} -s 192.168.1.0/255.255.255.0 -j ACCEPT
iptables -I FORWARD -i ${LOCAL} -d 192.168.0.0/255.255.255.0 -j DROP
iptables -A FORWARD -i ${LOCAL} -s 192.168.0.0/255.255.255.0 -j ACCEPT
iptables -A FORWARD -i ${DSL} -d 192.168.0.0/255.255.255.0 -j ACCEPT
iptables -A FORWARD -i ${CABLE} -d 192.168.0.0/255.255.255.0 -j ACCEPT
iptables -A FORWARD -i ${DSL} -d 192.168.1.0/255.255.255.0 -j ACCEPT
iptables -A FORWARD -i ${CABLE} -d 192.168.1.0/255.255.255.0 -j ACCEPT
iptables -t nat -A POSTROUTING -o ${DSL} -j SNAT --to-source 71.39.157.170
iptables -t nat -A POSTROUTING -o ${CABLE} -j SNAT --to-source 192.168.100.2
#This is the second part of what I thaught made this all work, it adds the mark 80 to the [packet/frame/?] - can someone tell me what exactly I am marking?
iptables -t mangle -A POSTROUTING -s 71.39.157.170/255.255.255.248 -j MARK --set-mark 80
iptables -t mangle -A POSTROUTING -s 192.168.100.2/255.255.255.0 -j MARK --set-mark 80
#Tell the kernel that ip forwarding is OK
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 1 > $f ; done
ip route + ip tables + 2 lan ip's + 2 isps only 1 issue!
