Masq/Forwarding with 3 NICs

oaf · 09-30-2007, 03:09 PM

Hi

I'm running CentOS 5 on a dual Xeon server with five network cards. I want to use the machine as a firewall and have set it up to masq/forward and firewall as per the IP Masquerade HOWTO, so far so good:

eth0 = "external" card connected to internet (and firewalled)
eth1 = internal with IP of 192.168.0.1 (netmask 255.255.255.0)

eth1 is connected to a switch and is successfully forwarding packets for my internal 192.168.0.* network (computers on that use 192.168.0.1 as their gateway).

However I want to use eth2 (a third physical card in my server) to connect to another machine and also forward packets with the same functionality as eth1->eth0. I've assigned eth2 the IP address of 192.168.0.12 and connected it up but I can't get it to forward packets.

Here's the relevant snippet of my firewall script:

Code:

# FORWARD: Enable Forwarding and thus IPMASQ
#
echo "     - FWD: Allow all connections OUT and only existing/related IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED \
 -j ACCEPT

$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT

# Oaf for eth2
$IPTABLES -A FORWARD -i $INTIF2 -o $EXTIF -j ACCEPT

$IPTABLES -A FORWARD -i $EXTIF -o $INTIF2 -m state --state ESTABLISHED,RELATED \
 -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $INTIF2 -m state --state ESTABLISHED,RELATED \
 -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF2 -o $INTIF -m state --state ESTABLISHED,RELATED \
 -j ACCEPT

# Catch all rule, all other forwarding is denied and logged.
#
$IPTABLES -A FORWARD -j reject-and-log-it

echo "     - NAT: Enabling SNAT (MASQUERADE) functionality on $EXTIF"

INTIF = eth1, INTIF2 = eth2 and EXTIF = eth0

Am I wrong to have both NICs on the subnet? Or is the problem elsewhere?

Please let me know if you need more info and thanks for reading

Oaf

andrewdodsworth · 09-30-2007, 05:58 PM

Pretty sure that if you changed eth2 to say 192.168.2.x life would be easier - subnetting and working out the netmask involves (at least for me!) a lot of headscratching.

oaf · 10-01-2007, 02:42 AM

Hi Andrew

Thanks for your reply. I think you're probably right and had come to the same conclusion myself

I'll have a go with 192.168.2.x and see how I get on. Will post results to the forum.

Cheers

Oaf

oaf · 10-01-2007, 03:34 PM

Hi

So... I decided to make my second internal network (on eth2) a separate subnet starting 192.168.1.0 and rewrote my firewall rules. It ALMOST works, but not quite!

- the "server" with the NICs in it can ping everything (int/int2/ext)
- the machines on int can ping the server but NOT int2 machines
- the machines on int2 can ping the server but NOT int1 machines

Everyone can get to the internet (external) so forwarding is working.

Is it some kind of routing issue (well, yes... but how/what?) and do I fix it by changing my firewall or by configuring the routing table, or something else?

I've included as much info as I can below. Any comments on whether the firewall looks sensible/changes to make to finish the last part of this would be much appreciated

Here is the output of ifconfig (ext IP blanked with x.x.x.x)

Code:

eth0      Link encap:Ethernet  HWaddr 00:B0:D0:20:E5:4F
          inet addr:x.x.x.x  Bcast:255.255.255.255 Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2169439 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1433083 errors:0 dropped:0 overruns:0 carrier:0
          collisions:1790 txqueuelen:1000
          RX bytes:3102962411 (2.8 GiB)  TX bytes:88647610 (84.5 MiB)

eth1      Link encap:Ethernet  HWaddr 00:90:27:FC:90:B0
          inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5374956 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10010489 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:330970541 (315.6 MiB)  TX bytes:596826515 (569.1 MiB)

eth2      Link encap:Ethernet  HWaddr 00:90:27:FC:90:B1
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7912 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3669 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:929420 (907.6 KiB)  TX bytes:215114 (210.0 KiB)

...and route:

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 eth2
192.168.0.0     *               255.255.255.0   U     0      0        0 eth1
82.9.236.0      *               255.255.252.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth2
default         abc2-mexch-0-0- 0.0.0.0         UG    0      0        0 eth0

Here's the now firewall - again, based on (my understanding of) the HOWTO example.

Code:

#!/bin/sh
#
# rc.firewall-iptables-stronger
#
FWVER=0.88s

IPTABLES=/sbin/iptables
LSMOD=/sbin/lsmod
DEPMOD=/sbin/depmod
MODPROBE=/sbin/modprobe
GREP=/bin/grep
AWK=/bin/awk
IFCONFIG=/sbin/ifconfig

#  For this example, "eth0" is external and "eth1+2" are internal
EXTIF="eth0"
INTIF="eth1"
INTIF2="eth2"

echo "  External Interface:  $EXTIF"
echo "  Internal Interface:  $INTIF"
echo "  2nd Intl Interface:  $INTIF2"
echo "  ---"

# Determine the external IP automatically:
EXTIP="`$IFCONFIG $EXTIF | $AWK \
 /$EXTIF/'{next}//{split($0,a,":");split(a[2],a," ");print a[1];exit}'`"

echo "  External IP: $EXTIP"
echo "  ---"

# Assign the internal TCP/IP network and IP address
INTNET="192.168.0.0/24"
INTNET2="192.168.1.0/24"
INTIP="192.168.0.1/32"
INTIP2="192.168.1.1/32"

echo "  Internal Network:  $INTNET"
echo "  Internal IP:       $INTIP"
echo "  Internal Network2: $INTNET2"
echo "  Internal IP2:      $INTIP2"
echo "  ---"

# Setting a few other local variables
#
UNIVERSE="0.0.0.0/0"

# Need to verify that all modules have all required dependencies
#
echo "  - Verifying that all kernel modules are ok"
$DEPMOD -a

echo -en "    Loading kernel modules: "

# With the new IPTABLES code, the core MASQ functionality is now either
# modular or compiled into the kernel.  This HOWTO shows ALL IPTABLES
# options as MODULES.  If your kernel is compiled correctly, there is
# NO need to load the kernel modules manually.  
#Load the main body of the IPTABLES module - "ip_tables"
#  - Loaded automatically when the "iptables" command is invoked
#
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "ip_tables, "
#
#Verify the module isn't loaded.  If it is, skip it
#
if [ -z "` $LSMOD | $GREP ip_tables | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_tables
fi


#Load the IPTABLES filtering module - "iptable_filter" 
#
#  - Loaded automatically when filter policies are activated


#Load the stateful connection tracking framework - "ip_conntrack"
#
# The conntrack  module in itself does nothing without other specific 
# conntrack modules being loaded afterwards such as the "ip_conntrack_ftp"
# module
#
#  - This module is loaded automatically when MASQ functionality is 
#    enabled 
#
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "ip_conntrack, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_conntrack | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_conntrack
fi

#Load the FTP tracking mechanism for full FTP tracking
#
# Enabled by default -- insert a "#" on the next line to deactivate
#
echo -e "ip_conntrack_ftp, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_conntrack_ftp | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_conntrack_ftp
fi

#Load the general IPTABLES NAT code - "iptable_nat"
#  - Loaded automatically when MASQ functionality is turned on
# 
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "iptable_nat, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP iptable_nat | $AWK {'print $1'} `" ]; then
   $MODPROBE iptable_nat
fi

#Loads the FTP NAT functionality into the core IPTABLES code
# Required to support non-PASV FTP.
echo -e "ip_nat_ftp"
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_nat_ftp | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_nat_ftp
fi

echo "  ---"

#CRITICAL:  Enable IP forwarding since it is disabled by default since
echo "  Enabling forwarding.."
echo "1" > /proc/sys/net/ipv4/ip_forward

echo "  Enabling DynamicAddr.."
echo "1" > /proc/sys/net/ipv4/ip_dynaddr

echo "  ---"

#Clearing any previous configuration
#
#  Unless specified, the defaults for INPUT, OUTPUT, and FORWARD to DROP
#
#    You CANNOT change this to REJECT as it isn't a vaild policy setting.
#    If you want REJECT, you must explictly REJECT at the end of a giving 
#    INPUT, OUTPUT, or FORWARD chain
#
echo "  Clearing any existing rules and setting default policy to DROP.."
$IPTABLES -P INPUT DROP
$IPTABLES -F INPUT 
$IPTABLES -P OUTPUT DROP
$IPTABLES -F OUTPUT 
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD 
$IPTABLES -F -t nat

#Not needed and it will only load the unneeded kernel module
#
#$IPTABLES -F -t mangle

# Delete all User-specified chains
$IPTABLES -X

# Reset all IPTABLES counters
$IPTABLES -Z

#Configuring specific CHAINS for later use in the ruleset
#
#  NOTE:  Some users prefer to have their firewall silently
#         "DROP" packets while others prefer to use "REJECT"
#         to send ICMP error messages back to the remote 
#         machine.  The default is "REJECT" but feel free to
#         change this below.
#
# NOTE: Without the --log-level set to "info", every single
#       firewall hit will goto ALL vtys.  This is a very big
#       pain.
#
echo "  Creating a DROP chain.."
$IPTABLES -N reject-and-log-it
$IPTABLES -A reject-and-log-it -j LOG --log-level info 
$IPTABLES -A reject-and-log-it -j REJECT

echo -e "\n   - Loading INPUT rulesets"

#######################################################################
# INPUT: Incoming traffic from various interfaces.  All rulesets are 
#        already flushed and set to a default policy of DROP. 
# loopback interfaces are valid.
#
$IPTABLES -A INPUT -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT


# local interface, local machines, going anywhere is valid
#
$IPTABLES -A INPUT -i $INTIF -s $INTNET -d $UNIVERSE -j ACCEPT
#Oaf: second NIC:
$IPTABLES -A INPUT -i $INTIF2 -s $INTNET2 -d $UNIVERSE -j ACCEPT

# remote interface, claiming to be local machines, IP spoofing, get lost
$IPTABLES -A INPUT -i $EXTIF -s $INTNET -d $UNIVERSE -j reject-and-log-it

# external interface, from any source, for ICMP traffic is valid
$IPTABLES -A INPUT -i $EXTIF -p ICMP -s $UNIVERSE -d $EXTIP -j ACCEPT


# Allow any related traffic coming back to the MASQ server in.
#  STATEFULLY TRACKED
$IPTABLES -A INPUT -i $EXTIF -s $UNIVERSE -d $EXTIP -m state --state \
 ESTABLISHED,RELATED -j ACCEPT

#
# ----- End OPTIONAL INPUT Section -----

# Catch all rule, all other incoming is denied and logged. 
#
$IPTABLES -A INPUT -s $UNIVERSE -d $UNIVERSE -j reject-and-log-it


# ---------------------------------------------------------------------

echo -e "   - Loading OUTPUT rulesets"

#######################################################################
# OUTPUT: Outgoing traffic from various interfaces.  All rulesets are 
#         already flushed and set to a default policy of DROP. 
# Workaround bug in netfilter
# See http://www.netfilter.org/security/2002-04-02-icmp-dnat.html
#
$IPTABLES -A OUTPUT -m state -p icmp --state INVALID -j DROP

# loopback interface is valid.
#
$IPTABLES -A OUTPUT -o lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT

# local interfaces, any source going to local net is valid
#
$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET -j ACCEPT
# Oaf eth2 interface
$IPTABLES -A OUTPUT -o $INTIF2 -s $EXTIP -d $INTNET2 -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET2 -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF2 -s $EXTIP -d $INTNET -j ACCEPT

# local interface, MASQ server source going to the local net is valid
#
$IPTABLES -A OUTPUT -o $INTIF -s $INTIP -d $INTNET -j ACCEPT
# Oaf eth2 interface

$IPTABLES -A OUTPUT -o $INTIF2 -s $INTIP2 -d $INTNET2 -j ACCEPT
# outgoing to local net on remote interface, stuffed routing, deny
#
$IPTABLES -A OUTPUT -o $EXTIF -s $UNIVERSE -d $INTNET -j reject-and-log-it

# anything else outgoing on remote interface is valid
#
$IPTABLES -A OUTPUT -o $EXTIF -s $EXTIP -d $UNIVERSE -j ACCEPT

# ----- Begin OPTIONAL OUTPUT Section -----
#
# DHCPd - Enable the following lines if you run an INTERNAL DHCPd server
#         - Remove BOTH #s all the #s if you need this functionality.
#
#$IPTABLES -A OUTPUT -o $INTIF -p tcp -s $INTIP --sport 67 \
# -d 255.255.255.255 --dport 68 -j ACCEPT
#$IPTABLES -A OUTPUT -o $INTIF -p udp -s $INTIP --sport 67 \
# -d 255.255.255.255 --dport 68 -j ACCEPT

#
# ----- End OPTIONAL OUTPUT Section -----


# Catch all rule, all other outgoing is denied and logged. 
#
$IPTABLES -A OUTPUT -s $UNIVERSE -d $UNIVERSE -j reject-and-log-it


echo -e "   - Loading FORWARD rulesets"

#######################################################################
# FORWARD: Enable Forwarding and thus IPMASQ
#

# ----- Begin OPTIONAL FORWARD Section -----
#
#  Put PORTFW commands here
#
# ----- End OPTIONAL FORWARD Section -----


echo "     - FWD: Allow all connections OUT and only existing/related IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED \
 -j ACCEPT

$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT

#Oaf eth2
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF2 -m state --state ESTABLISHED,RELATED \
 -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF2 -o $EXTIF -j ACCEPT


# Catch all rule, all other forwarding is denied and logged. 
#
$IPTABLES -A FORWARD -j reject-and-log-it


echo "     - NAT: Enabling SNAT (MASQUERADE) functionality on $EXTIF"
#
#More liberal form
#$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
#
#Stricter form
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j SNAT --to $EXTIP


#######################################################################
echo -e "\nrc.firewall-iptables-stronger $FWVER done.\n"

THANKS

Oaf

andrewdodsworth · 10-02-2007, 02:59 AM

Had a quick read and I think it should be as simple as adding paired forwarding statements between your internal interfaces - if you want to restrict traffic direction eg from my setup

Code:

-A forward_int -s 192.168.2.0/255.255.255.0 -d 192.168.3.0/255.255.255.0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A forward_int -s 192.168.3.0/255.255.255.0 -d 192.168.2.0/255.255.255.0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT

which gives one way access from '2' to '3' network. If you want symmetrical unrestricted access then just drop the state stuff.

Only other comment on your firewall script is that all the loopback stuff generally speaking should be the first stuff executed before anything other rules - just in case there's an error somewhere between your lo statements that kills everything.

oaf · 10-02-2007, 12:58 PM

Hi

Thanks for persevering with this... I added your suggestions (in red, with the IPs changed to match my setup):

Code:

# Allow any related traffic coming back to the MASQ server in.
#  STATEFULLY TRACKED
$IPTABLES -A INPUT -i $EXTIF -s $UNIVERSE -d $EXTIP -m state --state \
 ESTABLISHED,RELATED -j ACCEPT

$IPTABLES -A forward_int -s 192.168.1.0/255.255.255.0 -d 192.168.2.0/255.255.255.0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A forward_int -s 192.168.2.0/255.255.255.0 -d 192.168.1.0/255.255.255.0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
#
# ----- End OPTIONAL INPUT Section -----

# Catch all rule, all other incoming is denied and logged. 
#
$IPTABLES -A INPUT -s $UNIVERSE -d $UNIVERSE -j reject-and-log-it

and the firewall script gives this output:

Code:

(stuff trimmed)
  Enabling forwarding..
  Enabling DynamicAddr..
  ---
  Clearing any existing rules and setting default policy to DROP..
iptables: Directory not empty
iptables: Bad built-in chain name
iptables: No chain/target/match by that name
  Creating a DROP chain..
iptables: Chain already exists

   - Loading INPUT rulesets
iptables: No chain/target/match by that name
iptables: No chain/target/match by that name
   - Loading OUTPUT rulesets
   - Loading FORWARD rulesets
(stuff trimmed)

I tried taking the state restrictions out too but no joy. My understanding of iptables syntax is still a little iffy unfortunately

Do you have any more ideas please?

Thanks again for your help

Oaf

andrewdodsworth · 10-02-2007, 04:08 PM

In my rules apart from having different networks which you spotted and changed there is also a reference to 'forward_int' which is a chain that doesn't exist in your rules. Therefore you just need to replace reference to that with 'FORWARD' as in:

Code:

-A FORWARD -s 192.168.1.0/255.255.255.0 -d 192.168.2.0/255.255.255.0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.2.0/255.255.255.0 -d 192.168.1.0/255.255.255.0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT

I'd recommend reading about firewalls - I found Linux Firewalls by Robert L Ziegler very useful as it builds up the iptables stuff bit by bit starting with simple stuff first before going on to forwarding and NAT. Probably some good free tutorials around if you Google or elsewhere on LQ.

oaf · 10-02-2007, 04:37 PM

Hi Andrew

Thanks for your reply. I should have noticed the FORWARD(_INT) - that's what happens when you try things in a rush, apologies.

So using FORWARD I still appear to be being firewalled, /var/log/messages says

Code:

Oct  2 22:29:50 myserver kernel: IN=eth1 OUT=eth2 SRC=192.168.0.2 DST=192.168.1.20 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=17370 DF PROTO=TCP SPT=4952 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0

when I try to access a webserver hosted on 192.168.1.20 from a client on 192.168.0.2 (the log is telling me that it has blocked it, isn't it?)

I did notice that I'd made another silly error saying that the networks were 192.168.1.* and 192.168.2.* when they're 192.168.1.* and 192.168.0.* but I've changed the IPs and your suggestion still doesn't seem to be doing it for me (the block above).

I'll find some time to read up on iptables etc a bit more; to be honest I set my last firewall up several years ago (with just two NICs, internal and internet) which was much easier (no passing between additional subnets!) and I haven't had reason to stay very current with this stuff (when it just works, you don't need to...)

I'll keep reading

but thanks very much for your help

Oaf

andrewdodsworth · 10-02-2007, 06:00 PM

Do your forwarding rules look like this now?

Code:

-A FORWARD -s 192.168.0.0/255.255.255.0 -d 192.168.1.0/255.255.255.0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.1.0/255.255.255.0 -d 192.168.0.0/255.255.255.0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT

If they do then you need to look at the order of your rules to see if something else is stopping it. The other thing you can do is add logging rules so that you can determine exactly where traffic is being dropped.

I use the iptables-save command to see the exact order of rules and to try and work out what's happening with the packets of interest. Forwarding should be straightforward as there's nothing changing on either the source or destination IP/port.

The other thing you can do is run

Code:

iptables -L FORWARD -v

which will list the rules for the selected chain (ie FORWARD) with numbers of packets and bytes. That should help with spotting where the packets are going.

Hope this helps.

oaf · 10-03-2007, 02:13 PM

Hi Andrew

I'd like to thank you very much for persisting with this and for your patience! I've sorted it out: "iptables -L FORWARD -v" was the key and allowed me to see what was going on properly... until then I was struggling to visualize what was happening to the data.

I've now tidied up my firewall script (using $VARS throughout for my IPs/subnets to keep it consistent/more manageable) and got port forwarding working to specific private IPs for downloading torrents etc too.

Not so hard in the end thanks to some great tips (and my being able to spend some proper time on it today helped too; always the case that things become clearer when you stand back from the problem a little!)

Thanks again for your help, much appreciated

Oaf

andrewdodsworth · 10-03-2007, 02:49 PM

Glad it's sorted and pleased to have been able to help. In the process I rediscovered a thing or two about my own setup which I've now added to my documentation!