iptable mangle + ip rule fwmark + masquerade lost packets

wgualla · 04-10-2012, 04:58 AM

Hi, can anyone help me determine why I lose packets?

I have configured multiple VPNs to different customers having subnet overlapping (client 1, subnet 192.168.0.0/24 via vpn tunnel on ppp0; client 2, subnet 192.168.0.0/24 via vpn tunnel on ppp1 ), my solution is to assign two different subnets on the lan interface (client 1 192.168.100.0/24, client 2 192.168.101.0/24), mark packets with iptables mangle and one routing table for each mark as follows:

# Customer # 1:
iptables -t nat -I POSTROUTING -o ppp0 -j MASQUERADE
iptables -t mangle -I PREROUTING -i eth0 \
-d 192.168.100.0/24 -j MARK --set-mark 100
ip rule add fwmark 100 table 100
ip route add default dev ppp0 table 100

# Customer # 2:
iptables -t nat -I POSTROUTING -o ppp1 -j MASQUERADE
iptables -t mangle -I PREROUTING -i eth0 \
-d 192.168.101.0/24 -j MARK --set-mark 101
ip rule add fwmark 101 table 101
ip route add default dev ppp1 table 101

# Note: ppp0 has ip addr 192.168.180.7
# ppp1 has ip addr 192.168.180.8

ok, so far, so good, BUT I LOSE PACKETS!!!!:

trying from my lan machine (192.168.33.152: ping 192.168.100.251) packets arrives to router; looking at output interface I can see request & reply echo packets, but somewhere on linux connection tracking system, reply packets lose (I can see echo request packets on eth0, request & replay on ppp0, but reply packet doesn't arrives output interface eth0)

NOTE: disabling ip rule fwmark and using default main route table, everything works fine.

# tcpdump -nn -q -i eth0 icmp
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:33:01.019693 IP 192.168.33.152 > 192.168.100.251: ICMP echo request, id 768, seq 38145, length 40
11:33:06.207066 IP 192.168.33.152 > 192.168.100.251: ICMP echo request, id 768, seq 38401, length 40

# tcpdump -nn -q -i ppp0
listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
11:56:18.942009 IP 192.168.180.7 > 192.168.0.251: ICMP echo request, id 768, seq 40449, length 40
11:56:18.994315 IP 192.168.0.251 > 192.168.180.7: ICMP echo reply, id 768, seq 40449, length 40
11:56:24.259976 IP 192.168.180.7 > 192.168.0.251: ICMP echo request, id 768, seq 40705, length 40
11:56:24.312380 IP 192.168.0.251 > 192.168.180.7: ICMP echo reply, id 768, seq 40705, length 40

some additional information:

#uname -ar
Linux router 2.6.27-17-generic #1 SMP Fri Mar 12 03:09:00 UTC 2010 i686 GNU/Linux

router:~# lsb_release -d
Description: Ubuntu 8.10

thank everyone in advance.

nikmit · 04-11-2012, 02:54 AM

See http://www.linuxquestions.org/questi...debian-936957/ and the current thread by donalbane about ppp default route. No solution yet but I think the discussions might be related.

Other than that if you provide a sketch of what your layout looks like it will make it easier to provide a relevant answer.

One thing that doesn't look right to me is at this point is how you mark the packets destined for one specific IP address and then ping a different IP. How is your NAT set up?
To set this up with overlapping networks you will need a) the gateway on the client side doing port address translation, so you can route all traffic to their network with a static route for one or a few /32 ip addresses or b) devise a fairly complex NAT solution on your router, possibly by doing SNAT on the client facing interfaces and then marking on the internet facing interface based on the NAT-ed source address.

Nik

wgualla · 04-11-2012, 03:43 AM

Hi Nik, thank for your answer.
You're right, I forgot include nat commands. Sorry. I have use iptables NETMAP to translate each subnet.
iptables -I PREROUTING -d 192.168.100.0/24 -i eth0 -j NETMAP --to 192.168.0.0/24
iptables -I PREROUTING -d 192.168.101.0/24 -i eth0 -j NETMAP --to 192.168.0.0/24

Let me explain the situation with a little schema:

Code:


	Quote:
	
	
		
			
				
+-----------------+
|                 |      ping 192.168.100.251 reply from ORION
| my pc           +----+ ping 192.168.101.251 reply from HERCULES
|192.168.33.152   |    |
|                 |    |        +----- AT THIS POINT echo reply has gone.
+-----------------+    |        |
                       |<-------+
                       |
    +------------------+-----------+ iptables -d 192.168.100.0/24 -> --set-mark 100      ###  PREROUTING ###
    |                 eth0         | iptables -d 192.168.101.0/24 -> --set-mark 101
    |             192.168.33.10    | iptables -mark 100 -d 192.168.100.0/24 -> netmap --to 192.168.0.0/24
    |                              | iptables -mark 101 -d 192.168.101.0/24 -> netmap --to 192.168.0.0/24
    |    ppp0            ppp1      | 
    |192.168.180.7  192.168.180.8  | iptables -t nat -o ppp0 -j MASQUERADE  ### POSTROUTING ###
    +-----+-------------------+----+ iptables -t nat -o ppp1 -j MASQUERADE 
          |                   |<---------------------+
          | table 100         | table 101            |
          | fwmark 100        | fwmark 101           |
          | default dev ppp0  | default dev ppp1     +--  AT THIS POINT I'VE echo request & reply !!!!
          |                   |
          |                   |
          |                   |
 +--------+--------+  +-------+---------+
 |      pppX       |  |      pppY       |
 | 192.168.180.107 |  | 192.168.180.108 |
 |                 |  |                 |
 |  192.168.0.1    |  |  192.168.0.1    |
 |  eth0           |  |  eth0           |
 +---+-------------+  +---+-------------+
     |                    |
     |                    |
     |   ORION            |   HERCULES
     | +-------------+    | +-------------+
     +-+192.168.0.251|    +-+192.168.0.251|
     | +-------------+    | +-------------+
     |                    |
     .                    .
     | +-------------+    | +-------------+
     +-+192.168.0.250|    +-+192.168.0.250|
     | +-------------+    | +-------------+
     | +-------------+    | +-------------+
     +-+192.168.0.249|    +-+192.168.0.249|
       +-------------+      +-------------+

I hope this be helpful, I look forward to your suggestions.

Thank again.

nikmit · 04-11-2012, 04:25 AM

I suppose you have no other way of doing the addressing, or it is not up to you. Everything would be much simpler if the subnets don't overlap.

If you have to stick with it, the easiest way would be to do what all ISPs do - provide an address at pppY and pppX and conigure it so all traffic is masqueraded to that address.

Code:

          |                   |
 +--------+--------+  +-------+---------+
 |      pppX       |  |      pppY       | iptables -t nat -A postrouting -o pppY -j MASQUERADE
 | 192.168.180.107 |  | 192.168.180.108 | iptables -t nat -A postrouting -o pppX -j MASQUERADE
 |                 |  |                 |
 |  192.168.0.1    |  |  192.168.0.1    |
 |  eth0           |  |  eth0           |
 +---+-------------+  +---+-------------+
     |                    |

At the moment you have all 4 addresses in your 2 ppp pairs in the same 192.168.180.0/24 subnet, so the kernel sees them as connected to the same network.
Either reduce the subnet size, or change the network number on one of them. So pppX becomes 192.168.181.107 and ppp0 192.168.181.7

Code:

          |                   |
 +--------+--------+  +-------+---------+
 |      pppX       |  |      pppY       | iptables -t nat -A postrouting -o pppY -j MASQUERADE
 | 192.168.181.107 |  | 192.168.180.108 | iptables -t nat -A postrouting -o pppX -j MASQUERADE
 |                 |  |                 |
 |  192.168.0.1    |  |  192.168.0.1    |
 |  eth0           |  |  eth0           |
 +---+-------------+  +---+-------------+
     |                    |

I would expect things to work with just the above, and no further marking or NATing.

Your entire diagram would be

Code:

|                 |  default gateway 192.168.33.10    
| my pc           +----+ 
|192.168.33.152   |    |
|                 |    | 
+-----------------+    | 
                       |
                       |
    +------------------+-----------+ 
    |                 eth0         | 
    |             192.168.33.10    | 
    |                              | Once ppp0 and ppp1 are in different subnets, and orion and hercules NATed,
    |    ppp0            ppp1      | nothing should be needed here other than basic forwarding permissions.
    |192.168.181.7  192.168.180.8  | 
    +-----+-------------------+----+ 
          |                   |
          |                   | 
          |                   |
   box1   |                   | box2
 +--------+--------+  +-------+---------+
 |      pppX       |  |      pppY       | iptables -t nat -A postrouting -o pppY -j MASQUERADE (on box2)
 | 192.168.181.107 |  | 192.168.180.108 | iptables -t nat -A postrouting -o pppX -j MASQUERADE (on box1)
 |                 |  |                 |
 |  192.168.0.1    |  |  192.168.0.1    |
 |  eth0           |  |  eth0           |
 +---+-------------+  +---+-------------+
     |                    |
     |                    |
     |   ORION            |   HERCULES    default gateway 192.168.0.1 (for all hosts in both networks)
     | +-------------+    | +-------------+
     +-+192.168.0.251|    +-+192.168.0.251|
     | +-------------+    | +-------------+
     |                    |
     .                    .
     | +-------------+    | +-------------+
     +-+192.168.0.250|    +-+192.168.0.250|
     | +-------------+    | +-------------+
     | +-------------+    | +-------------+
     +-+192.168.0.249|    +-+192.168.0.249|
       +-------------+      +-------------+

wgualla · 04-12-2012, 05:33 PM

Hi Nik, thank again for your reply.

your suggestion would be fine if I wanted to go from Hercules to my pc but the reality is just the opposite.

Hercules is a TerminalServer and I would like to logme in on it (the same is true for Orion, and a lot of other machines, this is just an example).

All my clients have they own ip range selection and subnet definition, I've no way to change it (this is the root of the challenge, they were here for a long time before).

Beyond the theological issues of the case, my problem is that somewhere in the nine circles of connection tracking system my packages are lost on their way back and I suspect it is by the use of different routing tables of the main. (the problem persists even removing all the configuration settings relative to the box2/pppY)

if you still have temerity to continue dealing with this issue, I look forward to your comments.

Thanks again.

nikmit · 04-13-2012, 05:35 AM

OK I think I know what the task is now, and I got it working in a test set up. Hopefully this will work for you:

Code:

iptables -t mangle -A PREROUTING -i eth0 -d 192.168.100.0/24 -j MARK --set-mark 0x1111
iptables -t mangle -A PREROUTING -i eth0 -d 192.168.101.0/24 -j MARK --set-mark 0x2222
iptables -t nat -A PREROUTING -i eth0 -d 192.168.100.0/24 -j NETMAP --to 192.168.0.0/24
iptables -t nat -A PREROUTING -i eth0 -d 192.168.101.0/24 -j NETMAP --to 192.168.0.0/24
ip rule add fwmark 0x1111 table 2
ip rule add fwmark 0x2222 table 3
ip route add 0.0.0.0/0 dev ppp0 table 2             # if needed you can make the route more specific here
ip route add 0.0.0.0/0 dev ppp1 table 3             # and here
ip route flush cache

This is working with icmp on ethernet rather than ppp interfaces. There is a thread here describing problems with default routes and ppp interfaces, see how you get on and post

wgualla · 04-13-2012, 05:58 AM

Thank you again Nik, all you write is correct and works fine... theorically, but we had two day working around. The key point was to disable the rp_filter (Reverse Path filtering).

SOLVED AT LAST!!!!!!!!!!

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter