Packets go out one tunnel, come back another, then are dropped

Atomicmongoose · 12-05-2010, 08:19 PM

I’ve run into a of a routing issue pertaining to packets leaving a firewall, traversing and IPSec tunnel, hitting the target and then returning via a different tunnel, finally arriving back on the source firewall but on a different interface from where it started. Once the packet has returned to the firewall it is dropped… I’ve been unable to discover the reason for the drop.

Two sides to the system, Firewall A and Firewall B. Each firewall provides the default gateway to its respective side and offers a backup IPSec tunnel to the high capacity tunnel handled internally. The Layer 3 Switch uses OSPF and takes care of the bulk of the behind the scenes routing between the sides. In case of failure the Layer 3 switches direct traffic to use the Firewall tunnels to route traffic.
Firewall A Hosts PPTPD services and ideally PPTP clients should be able to ping anywhere in the system. Currently PPTP clients can ping anything on Firewall A’s side and up to the Layer 3 Switch on Firewall B

Code:

Diagram:

Firewall A  ------------Ipsec Tunnel------------Firewall B
(10.1.1.1/24)					(10.2.2.1/24)
PPTPD 						        |
(10.1.1.100-200)---PPTP Clients		                |
|							|
|							|
Layer 3 Switch ------------Fiber Tunnel---------Layer 3 Switch
(10.1.1.2)				         (10.2.2.2)
|							|
(10.1.2.0/24)					(10.2.3.0/24)
(10.1.3.0/24)					(10.2.4.0/24)
(10.1.4.0/24)					(10.2.5.0/24)

Any part of this diagram can ping any other part of the diagram, except the PPTP clients.

A packet from 10.1.1.100, (a pptp client somewhere in the world) attempting to reach 10.2.5.1 (A server on Firewall B’s Side) will travel to Firewall B over the Ipsec Tunnel, down to the B sides Layer 3, into the various gadgets to reach the target. The reply packet returns to the Layer 3 Switch, where the reply takes the Fiber tunnel to A Side. The packet then appears on Eth0 of Firewall A and … disappears somewhere in either the IPTABLES, Strongswan, or routing rules of the firewall.

I’ve tried several things:
-Accepting any packet from anywhere and various other IPTABLES rules. I’m confident the packet isn’t being dropped in the firewall rules.
-Forcing traffic from PPTP clients to use the layer 3 switches as the default gateways. I should note that this does work when I shutdown IPSEC. When IPSEC is turned back on something must happen in the routing tables that undercuts my specified route
-I’ve set rp_filter to 0
Any thoughts on further options?

4play · 12-06-2010, 05:52 PM

since the response comes back on the wrong interface the initial connection never gets it response and it will just timeout on the client rather then the firewall.

From what I understand of ipsec it operates on level 2 so your routing changes wont make a difference. (though im still learning about ipsec so I might be wrong)

Whats the ipsec tunnel for? I would much rather have no encryption overhead and try to make the layer 3 switches you have as redundant as possible.

stress_junkie · 12-06-2010, 06:35 PM

Given these two facts about IPSEC:

1) IPSEC can use either the real NIC in the computer or it can use a virtual NIC
2) IPSEC uses separate inbound channel and outbound channel to communicate with another host

is it possible that the inbound channel is bound to the physical NIC and the outbound channel is bound to a virtual NIC (or vice versa)

and

the physical NIC has different routing configuration than the virtual NIC?