How can I forward multiple public addresses with iptables?

Revenge282 · 06-12-2011, 08:31 PM

Hello everyone,

I hope that someone here can help me out because I have been stuck in a pickle the last few days. With my current setup, I have 2 servers, one Windows and the other Linux. I want to use the Linux box to act as a firewall for the Windows. I have 5 public IP addresses, which each need to be forwarded to a specific LAN address. I have tried many combinations of things I have come across, the closest thing I can get is the packets are forwarded to Windows, but Windows tries to reply to the original source of the packet rather than the Linux box, causing timeouts. I am currently running a fresh install of CentOS 5.6. To make things a bit clearer, since I am not very good with wording, I made this diagram: http://www.sniperboys.com/repo/thegoal.jpg

If anyone can help, I would greatly appreciate it!

Thanks!

Noway2 · 06-13-2011, 08:37 AM

Iptables should be able to perform the functions you need. In addition to the firewall chains (input, output, forward) there are also the NAT tables, prerouting and postrouting. I admit that I am not really an expert on configuring iptables for this purpose so I would have to refer you to one of the many how-to documents. In this particular case, I think you need to make use of the masquerade feature of iptables on your Linux machine. This will allow it to pass traffic between your eth0 and eth1 ports and it will convert between the address domains automatically. For example, I use my cell phone as a PPP modem and then connect devices to the Ethernet eth0 and use the Linux machine as the gateway between them (search for how to docs on sharing your wireless ethernet for several references). To do this, you also need to turn on ipv4 forwarding in the kernel.

ambrop7 · 06-14-2011, 03:58 PM

Something like this (but see alternative without NAT below):

Windows box:

To the single network interface, assign the four IP addresses 10.4.17.2/24, 10.4.17.3/24, 10.4.17.4/24, 10.4.17.5/24, and default gateway 10.4.17.1.

Linux box:

1. remove all firewall rules

2. Configure eth1 (adjust prefix /24 accordingly)
ip addr add dev eth1 208.115.216.46/24 # this must be first so other addresses are made "secondary", and we use this one for local traffic
ip addr add dev eth1 208.115.216.42/24
ip addr add dev eth1 208.115.216.43/24
ip addr add dev eth1 208.115.216.44/24
ip addr add dev eth1 208.115.216.45/24
ifconfig eth1 up

3. Configure default gateway (adjust address).
route add default gw 208.115.216.1

4. Now make sure you can access the Internet from the Linux box, and make sure (with e.g. Wireshark) that the source address of outgoing packets is 208.115.216.46.

5. Configure eth0.
ifconfig eth0 10.4.17.1/24

6. Now make sure the Windows box can ping 10.4.17.1 (or that the Linux box can ping the Windows box).

7. Enable IP forwarding.
echo 1 > /proc/sys/net/ipv4/ip_forward

8. Configure NAT for address 2
iptables -t nat -A POSTROUTING -o eth1 -s 10.4.17.2 -j SNAT --to-source 208.115.216.42
iptables -t nat -A PREROUTING -i eth1 -d 208.115.216.42 -j DNAT --to-destination 10.4.17.2

9. Configure NAT for address 3
iptables -t nat -A POSTROUTING -o eth1 -s 10.4.17.3 -j SNAT --to-source 208.115.216.43
iptables -t nat -A PREROUTING -i eth1 -d 208.115.216.43 -j DNAT --to-destination 10.4.17.3

10. Configure NAT for address 4
iptables -t nat -A POSTROUTING -o eth1 -s 10.4.17.4 -j SNAT --to-source 208.115.216.44
iptables -t nat -A PREROUTING -i eth1 -d 208.115.216.44 -j DNAT --to-destination 10.4.17.4

11. Configure NAT for address 5
iptables -t nat -A POSTROUTING -o eth1 -s 10.4.17.5 -j SNAT --to-source 208.115.216.45
iptables -t nat -A PREROUTING -i eth1 -d 208.115.216.45 -j DNAT --to-destination 10.4.17.5

12. From the Windows box, make sure you can access the Internet via one of the four addresses. One the Linux box, monitor with Wireshark both eth0 and eth1, and make sure that the local and remote addresses are being translated correctly. For example, find one (same) outgoing packet in both captures, and make sure the destination addresses correspond (e.g. by being 10.4.17.4 and 208.115.216.44 in the eth0 and eth1 capture, respectively).

13. Make sure that incoming connections from the Internet to one of the four Internet addresses are being forwarded to the appropriate local address of the Windows box.

-- alternative configuration without NAT --

This should also work. It avoids NAT and does not use any internal addresses - the Linux box simply forwards packets between the Internet and the Windows box, and the Windows box has the Internet addresses assigned.

Windows box:

To the single network interface, assign the four IP internet addresses 208.115.216.42/24, 208.115.216.43/24, 208.115.216.44/24, 208.115.216.45/24 (adjust netmask), and default gateway 208.115.216.1 (adjust gateway).

Linux box:

1. remove all firewall rules

2. Configure eth1.
ifconfig eth1 208.115.216.46/24 # adjust netmask

3. Configure default gateway.
route add default gw 208.115.216.1 # adjust address

4. Make sure the Linux box can access the Internet.

5. Enable IP forwarding.
echo 1 > /proc/sys/net/ipv4/ip_forward

6. Configure eth0.
ifconfig eth0 up

7. Add routes to server addresses through eth0.
route add -host 208.115.216.42 dev eth0
route add -host 208.115.216.43 dev eth0
route add -host 208.115.216.44 dev eth0
route add -host 208.115.216.45 dev eth0

8. Make sure the Windows box can ping the Linux box at 208.115.216.46, and any Internet address.

Revenge282 · 06-16-2011, 12:22 AM

I tried the first set of instructions. That led me to the Windows server receiving the packets, but trying to reply to the original source address rather than the Linux box.

I have not yet tried the second box because of the fact that both servers have their own block of IP addresses for the public, and for whatever reason, it will cost us $40 to have those switched. For the sake of our clients, we have to keep the addresses the same as before. Do you think that the second method will yield better results?

Noway2 · 06-16-2011, 04:53 AM

Have you configured the windows boxes to use your Linux (router) as the gateway? In other words, do they know that they need to route the traffic to that box or do they think they are on the Internet with your public IP? In the end, the traffic is destined to the original source IP, NOT your Linux box but it needs to route through your linux box first. This is why the traffic the traffic is being addressed to the source IP.

This is why I suggested that you search and read for how to documents on sharing your Internet connection: such as this one: http://www.ubuntugeek.com/sharing-in...in-ubuntu.html You should also read some of the many how to documents on using your Linux machine as a router. I know from experience that this works and you can adapt it to your multiple IP scenario by adding source and destination addresses.

You should also understand the difference between masquerade and SNAT. Here is a link on that subject: http://webcache.googleusercontent.co...www.google.com

In order for this to work for you effectively and securely you need to learn what you are doing. This is why I suggested search terms instead of telling you what to do. With all due respect to ambrop7, who I know was trying to be helpful, just plugging in commands that someone tells you without understanding what they do can be dangerous.

Edit: Here is an excellent tutorial on Iptables, which you are using to perform your routing functions. I have pointed it specifically to the NAT section, though you will want to start at the top.

ambrop7 · 06-16-2011, 04:53 AM

Quote:

Originally Posted by Revenge282

I tried the first set of instructions. That led me to the Windows server receiving the packets, but trying to reply to the original source address rather than the Linux box.

The replies sure won't have the Linux box as a destination IP address. That's the way NAT works. Also, does the first configuration actually work?

This is an example what happens to a packet coming into the network that goes to the Windows box, and how a reply goes out:

Packet comes from the Internet to eth1:
- Source IP 1.2.3.4
- Destination IP 208.115.216.42

Linux box forwards this packet to eth0, NAT changes the destination address (and maybe port):
- Source IP 1.2.3.4
- Destination IP 10.4.17.2

Windows box receives this packet.

Windows box sends a reply back, which the Linux box receives on eth0:
- Source IP 10.4.17.2
- Destination IP 1.2.3.4

Linux box forwards this packet to eth1, NAT changes the source address (and maybe port):
- Source IP 208.115.216.42
- Destination IP 1.2.3.4

This is correct behaviour. Why would you want the reply to have the Linux box as a destination IP address? The replies already go through the Linux box because the destination MAC address if of the Linux box.

It is possible to NAT the source address too, but I don't really see why anyone would want to do that. This way, for any connection to the Windows server, it would appear as if the Linux server is connecting, and the Windows server won't really know who (IP address) is connecting.

Considering the second option: Why do you think it will cost anything? You don't have to change any external IP address or any hardware, you'd just be using the same addresses in a different way (unless changing the configuration of the Windows server costs you).

Revenge282 · 06-16-2011, 04:08 PM

@ambrop7
Sorry for my last post. I wrote that just before going to bed late last night, so I don't think I was too focused while writing it. Please see my explanation below for something that is actually a bit more coherent.

@Noway2
Thank you very much for the links, I really appreciate it. I also understand and respect your point of view on the situation that I should learn myself, and I have been doing my best for about 2-3 weeks before this thread. However, I am in a real crunch for time right now, and I cannot afford to have much more down time, which is why I came to these forums as a last resort. I hate to ask others for help because I feel like I am putting a burden on someone who does not need it, and that is why I have been hesitant to come here. I already had a set of rules very similar to those that ambrop7 had posted well before I started this thread, and I got the same result that I am getting now. Again, I fully understand that you are trying to quietly nudge me in the right directions, and I respect your point of view, but I am in a really tight pickle right now. Please don't take that the wrong way...

As for the issue I am seeing now, traffic does come into Windows, and it does leave Windows properly. However, when it reaches Linux, it is denied for some reason. I confirmed this by doing a traceroute to google.com on the Windows box with the current setup, and it showed a timeout where Linux should have replied. From what I can see, all policies should be set to ACCEPT by default, so I don't know why it isn't being accepted. I can't see anything coming through via Wireshark on Linux when the traceroute hits Linux either.

Here is the traceroute from Windows to google.com with the current setup:
(Hop #4 should be the Linux server. This was confirmed by myself and the technicians at the datacenter.)

Code:

tracert google.com

Tracing route to google.com [74.125.227.48] over a maximum of 30 hops:

1    <1 ms     1 ms     2 ms  10.4.7.197
2    <1 ms    <1 ms    <1 ms  vl124.cr01-74.core1.dllstx3.dallas-idc.com [208.115.251.89]
3    <1 ms    <1 ms    <1 ms  te3-3.core1.bdr2.dllstx3.dallas-idc.com [208.115.192.57]
4     *        *        *     Request timed out.

Here is my iptables file for reference:

Code:

# Generated by iptables-save v1.3.5 on Thu Jun 16 14:58:35 2011
*filter
:INPUT ACCEPT [11283:1277664]
:FORWARD ACCEPT [24338:1122487]
:OUTPUT ACCEPT [11329:1447524]
[7:307] -A FORWARD -j ACCEPT 
COMMIT
# Completed on Thu Jun 16 14:58:35 2011
# Generated by iptables-save v1.3.5 on Thu Jun 16 14:58:35 2011
*nat
:PREROUTING ACCEPT [3239:177057]
:POSTROUTING ACCEPT [12203:671297]
:OUTPUT ACCEPT [1559:118511]
[12:636] -A PREROUTING -d 208.115.216.42 -i eth1 -j DNAT --to-destination 10.4.17.2 
[0:0] -A POSTROUTING -s 10.4.17.2 -o eth1 -j SNAT --to-source 208.115.216.42 
COMMIT
# Completed on Thu Jun 16 14:58:35 2011

ambrop7 · 06-16-2011, 04:14 PM

Please paste the following:

- iptables -t filter --list --verbose
- iptables -t mangle --list --verbose
- iptables -t nat --list --verbose
- route -n
- ip addr show
- ip rule show

Noway2 · 06-16-2011, 04:16 PM

Quote:

As for the issue I am seeing now, traffic does come into Windows, and it does leave Windows properly. However, when it reaches Linux, it is denied for some reason.

The first thing that comes to mind, is your firewall, which I agree it looks like your firewall should not be the issue.
The second thing is that you need to have ip forwarding enabled. The instructions given in the above posts have you write this to the /proc file, which is not persistent. If you have performed any sort of reboot, logoff, etc, without rewriting this feature, that may be your issue. There are ways to make it persistent, you effectively have to put it in a startup script file, the best way depends on your distribution.

frieza · 06-16-2011, 04:29 PM

i have actually done this with a linksys router runing dd-wrt
another way of doing this is creating 'virtual' interfaces

eth0 208.115.216.2/24 gateway 208.115.216.1
eth0:1 208.115.216.3/24
eth0:2 208.115.216.4/24
...
eth0:n 208.115.216.n/24

then use iptables to do the rest of the dirty work as posted by ambrop7

Revenge282 · 06-16-2011, 05:09 PM

Thank you to everyone for your replies!

@ambrop7
iptables -t filter --list --verbose

Code:

Chain INPUT (policy ACCEPT 930 packets, 73550 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 96 packets, 4394 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 944 packets, 134K bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables -t mangle --list --verbose

Code:

Chain PREROUTING (policy ACCEPT 1592 packets, 115K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1188 packets, 96107 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 404 packets, 18670 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1186 packets, 169K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1590 packets, 188K bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables -t nat --list --verbose

Code:

Chain PREROUTING (policy ACCEPT 117 packets, 6372 bytes)
 pkts bytes target     prot opt in     out     source               destination
   17   901 DNAT       all  --  eth1   any     anywhere             D3872               to:10.4.17.2
   18   982 DNAT       all  --  eth1   any     anywhere             43-216-115-208.static.reverse.lstn.net to:10.4.17.3
    4   233 DNAT       all  --  eth1   any     anywhere             44-216-115-208.static.reverse.lstn.net to:10.4.17.4
    7   392 DNAT       all  --  eth1   any     anywhere             45-216-115-208.static.reverse.lstn.net to:10.4.17.5

Chain POSTROUTING (policy ACCEPT 91 packets, 5482 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  any    eth1    10.4.17.2            anywhere            to:208.115.216.42
    0     0 SNAT       all  --  any    eth1    10.4.17.3            anywhere            to:208.115.216.43
    0     0 SNAT       all  --  any    eth1    10.4.17.4            anywhere            to:208.115.216.44
    0     0 SNAT       all  --  any    eth1    10.4.17.5            anywhere            to:208.115.216.45

Chain OUTPUT (policy ACCEPT 31 packets, 2232 bytes)
 pkts bytes target     prot opt in     out     source               destination

route -n

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.4.2.128      0.0.0.0         255.255.255.252 U     0      0        0 eth0
208.115.216.40  0.0.0.0         255.255.255.248 U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
0.0.0.0         208.115.216.41  0.0.0.0         UG    0      0        0 eth1

ip addr show

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:1c:c0:f8:66:da brd ff:ff:ff:ff:ff:ff
    inet 208.115.216.42/29 brd 208.115.216.47 scope global eth1
    inet 208.115.216.43/29 brd 208.115.216.47 scope global secondary eth1:0
    inet 208.115.216.44/29 brd 208.115.216.47 scope global secondary eth1:1
    inet 208.115.216.45/29 brd 208.115.216.47 scope global secondary eth1:2
    inet 208.115.216.46/29 brd 208.115.216.47 scope global secondary eth1:3
    inet6 fe80::21c:c0ff:fef8:66da/64 scope link
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0a:cd:18:eb:63 brd ff:ff:ff:ff:ff:ff
    inet 10.4.2.130/30 brd 10.4.2.131 scope global eth0
    inet6 fe80::20a:cdff:fe18:eb63/64 scope link
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

ip rule show

Code:

0:      from all lookup 255
32766:  from all lookup main
32767:  from all lookup default

@Noway2
I am not sure by what you mean in your first sentence. As for the forwarding, I do have this at the top of /etc/sysctl.conf:

Code:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

sysctl -a also shows this is true after reboot.

@frieza
I was under the impression from what I read and was told that iptables does not support virtual interfaces. However, I could just as easily be wrong. I will look into this again and see if I get any different results.

Thanks again to all three of you!

ambrop7 · 06-16-2011, 06:02 PM

There's something weird with your eth0 address; it is 10.4.2.130/30, and then you have a manual gateway-less route 10.0.0.0/8 through eth0. Why don't you just assign the address 10.4.2.130/8 instead?
On the other hand - if there are routers (which appears from your traceroute) inside behind eth0, you should keep your 10.4.2.130/30 address, but add an actual gateway to the 10.0.0.0/8 route.

That traceroute you did looks bad to me. The two intermediate routers have Internet IP addresses (208.115.251.89, 208.115.192.57), rather than having private addresses in 10.0.0.0/8 - which everything behind eth0 should be. Could you please post here a Wireshark dump on eth0 of "ping www.google.com" from the Windows server?

Revenge282 · 06-16-2011, 06:43 PM

http://www.sniperboys.com/firewall/googleping.pcap

The ping results timed out to google.com.

As for the router situation, the tech at the datacenter told me that the hops work like this:

The switch that the Windows server is connected to
The "core" where I assume all switches are connected to
The switch that the Linux server is connected to
The actual Linux server
Everything after it leaves Linux

This is a traceroute from Linux to Windows:

Code:

traceroute to 10.4.17.2 (10.4.17.2), 30 hops max, 40 byte packets
 1  10.4.2.129 (10.4.2.129)  0.585 ms  0.812 ms  1.055 ms
 2  vl24.cr02-57.core1.dllstx3.dallas-idc.com (208.115.252.201)  2.785 ms  2.808 ms  2.839 ms
 3  ge0-1.vl124.cr01-74.dllstx3.dallas-idc.com (208.115.251.90)  78.766 ms  79.119 ms  79.473 ms
 4  10.4.17.2 (10.4.17.2)  0.376 ms  0.380 ms  0.377 ms

And from Windows to Linux:

Code:

Tracing route to 10.4.2.130 over a maximum of 30 hops

  1     1 ms     1 ms     2 ms  10.4.7.197
  2    <1 ms    <1 ms    <1 ms  vl124.cr01-74.core1.dllstx3.dallas-idc.com [208.115.251.89]
  3    <1 ms     2 ms     1 ms  ge0-1.vl24.cr02-57.dllstx3.dallas-idc.com [208.115.252.202]
  4    <1 ms    <1 ms    <1 ms  10.4.2.130

Trace complete.

ambrop7 · 06-16-2011, 07:28 PM

Ok, so it appears that packets are not being forwarded from eth0 to eth1. I see that the match counters are zero in the "iptables -t nat --list --verbose" output - but there should be a match for every packet that is forwarded from the Windows server to the Internet.

Please also post the eth0 dump of the same ping so we can confirm this.

I have not yet determined the cause of this. It definitely can't be iptables. It also cannot be a missing or wrong route (the default route seems fine). It also can't be the Reverse Path Filter (the source address of the pings is as expected, and the Linux box would route a reply back to eth0).

I have however noticed that the IP header checksum of all packets from 10.4.17.2 is wrong. This could very well be your problem. It's not just in the capture file - you can get bogus checksums in dumps for outgoing packets when checksum offloading is used, but these are received packets. So you better get that fixed.

Revenge282 · 06-16-2011, 09:22 PM

http://sniperboys.com/firewall/googlepingeth0.pcap

When you said the same ping and a dump on eth0, I assumed you meant to do a ping from Windows with a capture on Linux eth0. Correct me if I was wrong.

As for, the checksum issue. I am not to sure about how to go about fixing that. I was looking around and found something called "TCP Checksum Offload" and "UDP Checksum Offload", both were set to "Rx & Tx Enabled". From what I Googled, the error had something to do with checksum offloads. That is the best I can come up with if it has to do with Windows. As for Linux, I found something that uses ethtool to enable and disable checksum offloading (rx/tx on/off).