How can I forward multiple public addresses with iptables?

ambrop7 · 06-17-2011, 04:09 AM

Considering checksums, you can try:

Code:

ethtool -K eth0 tx off
ethtool -K eth0 rx off

So this isn't as I suggested:

Code:

    inet 208.115.216.42/29 brd 208.115.216.47 scope global eth1
    inet 208.115.216.43/29 brd 208.115.216.47 scope global secondary eth1:0
    inet 208.115.216.44/29 brd 208.115.216.47 scope global secondary eth1:1
    inet 208.115.216.45/29 brd 208.115.216.47 scope global secondary eth1:2
    inet 208.115.216.46/29 brd 208.115.216.47 scope global secondary eth1:3

The primary address on eth1 should be 208.115.216.46, not 208.115.216.42. I guess you ran the "ip" commands in the different order. But IMO this isn't the problem, it will just make the Linux box itself use the 208.115.216.42 address when accessing the Internet - but try changing it anyway.

I suggest that you start captures on eth0 and eth1, than try as many things as possible. For example, from both the Linux and the Windows box, try:
- ping www.google.com
- do some DNS queries (make sure try're uncached addresses)
- TCP to www.google.com:80

And from some system on the Internet, try:
- ping 208.115.216.46
- TCP to some open port on 208.115.216.46 (the Linux box is listening)
- ping 208.115.216.42
- TCP to some open port on 208.115.216.42 (the Windows box is listening)

What seems very strange is that DNS lookups on the Windows box seem to work, but not pings and TCP? From your routing table it appears that the DNS server is on the Internet, so I see no reason why DNS would be forwarded all right but nothing else.

ambrop7 · 06-17-2011, 04:14 AM

Also, the following may help debugging (then watch dmesg):

Code:

echo 1 > /proc/sys/net/netfilter/nf_conntrack_log_invalid
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
echo 1 > /proc/sys/net/ipv4/conf/eth0/log_martians

Revenge282 · 06-17-2011, 02:00 PM

The addresses have been switched as you said. As for the checksums, it tells me that the operation is not supported.

ethtool -k eth0

Code:

Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

ethtool -k eth1

Code:

Offload parameters for eth1:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off

When pinging google.com on Linux, the Wireshark dump showed, from what I could see, that there were no issues with pinging google.com. The eth0 capture did not show anything relevant from the ping.

As for pinging from Windows, the DNS was reachable if I specified it (as I had before) in the the private interface. If I disabled the DNS, it would not resolve anything. Either way, the packets did not make it to Linux via DNS, ICMP, TCP, or UDP. I did notice that the headers of the reply packets from Windows have bad checksums in Wireshark.

Same results apply for contacting the boxes from my home computer and from a friends.

Also, the first line of your debug suggestion:

Code:

echo 1 > /proc/sys/net/netfilter/nf_conntrack_log_invalid

returned that there was no file by that name. I was questioning myself wether or not to add it, but I decided to play it safe and wait for further word.

Thank you again for the reply, I really appreciate the help.

ambrop7 · 06-17-2011, 02:45 PM

Quote:

Originally Posted by Revenge282

Either way, the packets did not make it to Linux via DNS, ICMP, TCP, or UDP.

Are you sure? Because from what I've seen so far, packets from Windows are getting to Linux eth0, but Linux isn't forwarding them out through eth1. That is, any attempt of Windows to communicate with an external host will result in the outgoing packets to show up in the eth0 dump, but not in the eth1 dump. Could you confirm that?

I'm getting more and more sure that the checksum problem is in fact the real issue here. Because eth0 does not support checksum offloading (seen from ethtool output), the eth0 dumps are correct and the checksums really are wrong.

I suggest doing a simple test: pull out whatever is connected to eth0, and instead connect an unrelated computer directly to eth0, and give it the following configuration:
- IP address 10.4.17.2/8
- default gateway 10.4.2.130
- some DNS server on the Internet

This should be equivalent to your current setup, from the point of view of the Linux box. If the test box can access the Internet, there's nothing wrong with the configuration on the Linux box.

EDIT: the proc commands enable some logging. So check your dmesg for any messages about bad packets.

Revenge282 · 06-17-2011, 06:51 PM

The datacenter was unable to connect a laptop to the private interface of the Linux box. However, they confirmed that the box was pingable from multiple locations across the datacenter and that the switches connecting Windows to Linux should not be causing any issues.

I still cannot see any traffic from Windows reaching Linux on eth0 aside from some random packets. (10.4.17.2 sending traffic to 10.4.17.3) I believe that this traffic was from a game server control panel, judging by the port (8888) that it was destined for. But I guess that is to be expected if the gateway is set to Linux?

ambrop7 · 06-18-2011, 02:06 AM

Quote:

Originally Posted by Revenge282

However, they confirmed that the box was pingable from multiple locations

Quote:

Originally Posted by Revenge282

I still cannot see any traffic from Windows reaching Linux on eth0 aside from some random packets.

So, is the Linux box pingable from the Windows box? If if's pingable from other boxes but not the Windows box, you can try configuring it to forward traffic for one of the boxes that it's pingable for (that is, add the two iptables commands that do NAT for its address, and change the default gateway on that box). If you get it to forward traffic for another box, it means that the Linux box is configured just fine. Then check in Wireshark to see if packets from this working box have bad checksums. If they do not, than the checksums are the issue, and you can only fix your problem by fixing the checksums.

The only reason I could think of for bad checksums is a broken network interface somewhere between the Windows box (inclusive) and the Linux box (non-inclusive).

Revenge282 · 06-18-2011, 02:11 AM

Both boxes can ping each other. As for trying to forward elsewhere, I unfortunately do not have another server to try this with.

Do you have any idea about how I could go about fixing those checksums since the ethtool -K eth0 rx off and tx off were not "supported"? I have never run into an issue concerning checksums.

ambrop7 · 06-18-2011, 03:06 AM

So you're saying that:
- the Windows box can ping the Linux box, and
- packets from the Windows box to the Internet do not reach the Linux box?

If this is the case, there are routing problems in your network. I hope you understand that, on system A, when you set a route with a gateway address G (including default gateway), then G must be directly connected to A. If G is not directly connected (there are routers on the path), your should instead be specifying the first router (R1) on the path A->G, and add an appropriate route on R1. The same applies to R1 and all other routers (i.e. if the path is A->R1->R2->G, then A should have a route with gateway R1, R1 should have a route with gateway R2, and R2 should have a route with gateway G). This applies for the other direction as well (but with routes for A, not default route).

If this routing policy (A goes to the Internet through G) is specific to A, i.e. routers R1 and R2 should route Internet traffic from other computers through some other Internet connection (not the Linux box), than you have two options:

1. Add source-address-specific routes on R1 and R2.
2. If the above is not possible, your only way out may be to set up a VPN between A and G.

Revenge282 · 06-18-2011, 11:20 AM

I will try to setup a VPN first and get back to you. If I cannot get that to work, I will ask the datacenter if it is possible to have the two servers directly connected.

Thanks again for the reply!

ambrop7 · 06-18-2011, 12:31 PM

Quote:

Originally Posted by Revenge282

I will try to setup a VPN first and get back to you.

You can use my own VPN software for that, see http://code.google.com/p/badvpn/ .

Revenge282 · 06-19-2011, 12:26 PM

I notice that all traffic outbound from Windows has bad checksums, even on the public interface with the private disabled (and of course private with public disabled). The Director of Support from my datacenter spoke with me on the phone and he said he could for sure tell me that traffic was getting from Windows to Linux without any issues, and that he could safely say that the private network was not causing any of the problems since we saw through the traceroutes that the traffic did make it through all of the hops until it got to Linux. He suggest that there is a misconfiguration with the NAT setup somewhere. I have not yet tried to setup the VPN since I think that the checksum issue can be ruled out now that I see it happening on all interfaces with all outgoing traffic.

Again, here are the rules that 'service iptables status' returns:

Code:

Table: mangle
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Table: nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DNAT       all  --  0.0.0.0/0            208.115.216.42      to:10.4.17.2 
2    DNAT       all  --  0.0.0.0/0            208.115.216.43      to:10.4.17.3 
3    DNAT       all  --  0.0.0.0/0            208.115.216.44      to:10.4.17.4 
4    DNAT       all  --  0.0.0.0/0            208.115.216.45      to:10.4.17.5 

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    SNAT       all  --  10.4.17.2            0.0.0.0/0           to:208.115.216.42 
2    SNAT       all  --  10.4.17.3            0.0.0.0/0           to:208.115.216.43 
3    SNAT       all  --  10.4.17.4            0.0.0.0/0           to:208.115.216.44 
4    SNAT       all  --  10.4.17.5            0.0.0.0/0           to:208.115.216.45 

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Here is the iptables file:

Code:

# Generated by iptables-save v1.3.5 on Sun Jun 19 12:09:46 2011
*filter
:INPUT ACCEPT [2213487:671506563]
:FORWARD ACCEPT [202698:9291349]
:OUTPUT ACCEPT [2410365:1299045180]
COMMIT
# Completed on Sun Jun 19 12:09:46 2011
# Generated by iptables-save v1.3.5 on Sun Jun 19 12:09:46 2011
*nat
:PREROUTING ACCEPT [6042:325841]
:POSTROUTING ACCEPT [51094:2816296]
:OUTPUT ACCEPT [4556:330219]
[20853:1102035] -A PREROUTING -d 208.115.216.42 -i eth1 -j DNAT --to-destination 10.4.17.2 
[15946:853060] -A PREROUTING -d 208.115.216.43 -i eth1 -j DNAT --to-destination 10.4.17.3 
[3313:182944] -A PREROUTING -d 208.115.216.44 -i eth1 -j DNAT --to-destination 10.4.17.4 
[6413:347349] -A PREROUTING -d 208.115.216.45 -i eth1 -j DNAT --to-destination 10.4.17.5 
[0:0] -A POSTROUTING -s 10.4.17.2 -o eth1 -j SNAT --to-source 208.115.216.42 
[0:0] -A POSTROUTING -s 10.4.17.3 -o eth1 -j SNAT --to-source 208.115.216.43 
[0:0] -A POSTROUTING -s 10.4.17.4 -o eth1 -j SNAT --to-source 208.115.216.44 
[0:0] -A POSTROUTING -s 10.4.17.5 -o eth1 -j SNAT --to-source 208.115.216.45 
COMMIT
# Completed on Sun Jun 19 12:09:46 2011
# Generated by iptables-save v1.3.5 on Sun Jun 19 12:09:46 2011
*mangle
:PREROUTING ACCEPT [2416647:680829004]
:INPUT ACCEPT [2213681:671523900]
:FORWARD ACCEPT [202957:9303317]
:OUTPUT ACCEPT [2410549:1299075061]
:POSTROUTING ACCEPT [2613582:1308392426]
COMMIT
# Completed on Sun Jun 19 12:09:46 2011

Thanks again for the help!

ambrop7 · 06-19-2011, 12:46 PM

All right, so the checksums are fine, and I have no idea why they appear bad (if they were really bad, you wouldn't be able to ping).

Please re-read my post above where I explain why your routing setup is bad.

As far as I understand, this is what I think is happening (but I'm not completely sure):

- Windows wants to send a packet to some Internet address.
- Windows looks up the routing table and determines that Linux will be the gateway.
- Windows sees that Linux isn't directly connected. So it looks up Linux in the routing table, and finds a route to Linux through Router.
- Windows sends the packet out: destination IP = Linux, destination MAC = Router.
- Router receives the packet.
- Router looks up the destination IP (Linux). But Router's default gateway is something other than Linux - so it forwards it out there.
- (Linux never receives the packet.)

However, ping Linux from Windows works well, because Router has a correct route to Linux.

So, you need to be careful when saying whether "traffic is getting from Windows to Linux" - because some traffic may be, but some may not be getting through.

If you don't believe me, just try the virtual machine test I suggested earlier.

ambrop7 · 06-19-2011, 02:04 PM

This is all too weird...

- The checksums are bad, but ping works !?
- You posted two dumps of eth0. In one, packets from Windows towards the Internet are present, but not in the other one. Why - what was different?

I suggest you try to clear up your knowledge of the situation. Preferably, by analyzing yourself and getting reliable data rather than from other people.

Now I have two hypotheses of the problem:
1. Routing problems, as described above. Test this by capturing eth0, and pinging from the Windows box the Linux box itself (208.115.216.42) and an external Internet address. If both show up in the capture, routing is fine.
1. Checksums. If above, ping 208.115.216.42 (Linux) succeeded and its checksum was right, but ping Internet failed and its checksum was wrong, than checksums are definitely the reason for packets being dropped.