Multi-WAN setup not load balancing / no round robin (iptables, ip routes, ip rules)

robopt · 07-01-2013, 08:05 PM

I tried posting this on FedoraForum over the weekend, but it has yet to get any replies, so I figure it wouldn't hurt to branch out. Additionally I believe my previous attempt was filtered due to the LACP link I had in it.

I have been bashing my head on this for a few hours now and I have seen to have exhausted most of my Google and forum search results. Server can connect to both gateways and properly performs NAT/Masquerade for a client. The server doesn't seem to be load balancing between the two connections for said client. I just can't figure out the last piece to this puzzle and could use some help.

Overview:
- 2 ISP's -> Fedora Server -> LAN
- 2 ISP's both of same "weight"
- 1 Fedora 18 server with 3 NIC's installed, 2 external facing and 1 internal facing
- Dhcp, ip rules, iptables, and ip routes used

Notes:
192.168.0.1 = ISP0 Gateway
192.168.0.10 = wan0 (interface facing external ISP0)
192.168.1.1 = ISP1 Gateway
192.168.1.10 = wan1 (interface facing external ISP1)
192.168.3.65 = lan0 (interface facing internal LAN)
T0 (10) = routing table for wan0/isp0
T1 (20) = routing table for wan1/isp1

Connection 1 (1 minute of browsing, torrents, chat clients):

Code:

TX:             cum:    914kB   peak:   3.57Mb                                                                                      rates:   3.57Mb  3.57Mb  3.57Mb
RX:                    33.7kB            135kb                                                                                                135kb   135kb   135kb
TOTAL:                  948kB           3.70Mb                                                                                               3.70Mb  3.70Mb  3.70Mb

Connection 2 (1 minute of browsing, torrents, chat clients):

Code:

TX:             cum:      0B    peak:      0b                                                                                          rates:      0b      0b      0b
RX:                    2.35kB           1.23kb                                                                                                     0b      0b    219b
TOTAL:                 2.35kB           1.23kb                                                                                                     0b      0b    219b

ip tables

Code:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
10 T0 #eth0
20 T1 #eth1

Config

Code:

ip route add 192.168.0.0/24 dev wan0 src 192.168.0.10 table T0
ip route add default via 192.168.0.1 table T0
ip route add 192.168.1.0/24 dev wan1 src 192.168.1.10 table T1
ip route add default via 192.168.1.1 table T1
ip route add 192.168.0.0/24 dev wan0 src 192.168.0.10
ip route add 192.168.1.0/24 dev wan1 src 192.168.1.10
ip route del default
ip route add default scope global nexthop via 192.168.0.1 dev wan0 weight 1 nexthop via 192.168.1.1 dev wan1 weight 1
ip rule add from 192.168.0.10 table T0
ip rule add from 192.168.1.10 table T1
ip rule add to 8.8.8.8 table T0
ip rule add to 8.8.8.8 table T1
iptables -A POSTROUTING -t mangle -j CONNMARK --restore-mark
iptables -A POSTROUTING -t mangle -m mark ! --mark 0 -j ACCEPT
iptables -A POSTROUTING -p tcp --dport 21 -t mangle -j MARK --set-mark 10
iptables -A POSTROUTING -p tcp --dport 80 -t mangle -j MARK --set-mark 20
iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark
iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
iptables -A POSTROUTING -t mangle -m mark  --mark 10 -j ACCEPT
iptables -A POSTROUTING -t mangle -m mark  --mark 20 -j ACCEPT
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A PREROUTING -j MARK --set-mark 10
iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
iptables -t mangle -A PREROUTING -j CONNMARK --save-mark
iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o wan1 -j MASQUERADE
iptables -A FORWARD -i lan0 -j ACCEPT
iptables -A FORWARD -o lan0 -j ACCEPT

iptables mangle
- please note some port #'s were tests to see if that was the issue

Code:

*mangle
:PREROUTING ACCEPT [120:33352]
:INPUT ACCEPT [88:6242]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [44:4656]
:POSTROUTING ACCEPT [44:34656]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_direct - [0:0]
:PRE_ZONE_public - [0:0]
:PRE_ZONE_public_allow - [0:0]
:PRE_ZONE_public_deny - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -j CONNMARK --restore-mark
-A PREROUTING -m mark ! --mark 0 -j ACCEPT
-A PREROUTING -j MARK --set-mark 10
-A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
#-A PREROUTING -m state --state new -j MARK --set-mark 10
#-A PREROUTING -m state --state new -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
-A PREROUTING -j CONNMARK --save-mark
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -m mark  --mark 10 -j ACCEPT
-A POSTROUTING -m mark  --mark 20 -j ACCEPT
#-A POSTROUTING -j CONNMARK --restore-mark
#-A POSTROUTING -m mark ! --mark 0 -j ACCEPT
#-A POSTROUTING -m mark --mark 0 -p tcp --dport 21 -j MARK --set-mark 10
#-A POSTROUTING -m mark --mark 0 -p tcp --dport 80 -j MARK --set-mark 20
#-A POSTROUTING -m mark --mark 0 -p tcp -j MARK --set-mark 3
#-A POSTROUTING -j CONNMARK --save-mark
-A PREROUTING_ZONES -i eth0 -g PRE_ZONE_public
-A PREROUTING_ZONES -i eth2 -g PRE_ZONE_public
-A PREROUTING_ZONES -i eth1 -g PRE_ZONE_public
-A PREROUTING_ZONES -g PRE_ZONE_public
-A PRE_ZONE_public -j PRE_ZONE_public_deny
-A PRE_ZONE_public -j PRE_ZONE_public_allow
COMMIT

PREROUTING
I have tried commenting out the PREROUTING_direct/PREROUTING_ZONES in the iptables with no effect.

Code:

[user@localhost ~]$ sudo iptables -L PREROUTING -t mangle -v
Chain PREROUTING (policy ACCEPT 3586 packets, 317K bytes)
 pkts bytes target     prot opt in     out     source               destination
 107K  107M PREROUTING_direct  all  --  any    any     anywhere             anywhere
 107K  107M PREROUTING_ZONES  all  --  any    any     anywhere             anywhere
 107K  107M CONNMARK   all  --  any    any     anywhere             anywhere             CONNMARK restore
 104K  107M ACCEPT     all  --  any    any     anywhere             anywhere             mark match ! 0x0
 3589  317K MARK       all  --  any    any     anywhere             anywhere             MARK set 0xa
 1797  154K MARK       all  --  any    any     anywhere             anywhere             statistic mode random probability 0.50000000000 MARK set 0x14
 3589  317K CONNMARK   all  --  any    any     anywhere             anywhere             CONNMARK save

POSTROUTING
I have tried commenting out the POSTROUTING_direct in the iptables with no effect.

Code:

[user@localhost ~]$ sudo iptables -L POSTROUTING -t mangle -v
Chain POSTROUTING (policy ACCEPT 3273 packets, 674K bytes)
 pkts bytes target     prot opt in     out     source               destination
 168K  164M POSTROUTING_direct  all  --  any    any     anywhere             anywhere
59072   44M ACCEPT     all  --  any    any     anywhere             anywhere             mark match 0xa
 106K  120M ACCEPT     all  --  any    any     anywhere             anywhere             mark match 0x14

Any ideas, thoughts, and criticism is greatly appreciated. I am new to the Linux scene so I would love to be corrected if I am doing something wrong!