Howdy,

We've a huuuuuuuuuuuuuuuuuuuge problem in that a VPN we're trying to establish is failing the Phase 1 exchange apparently due to a consistently missing UDP fragment which is not being recieved by our firewall. It seems it's being lost in either our ISP or our Peer ISP's AS.

Any recommendations on wtf we can do to try and find the issue? I'd expect Backtrack would contain some decent tools to do a really fscking good UDP traceroute with a large payload between the two peer IP addresses. Any one know of the right tool for the job?

I don't know anything about TCP, but why the hell would it be using UDP for an exchange ?. Aren't lost packets "ho-hum" for UDP ?.

Does a 'tracepath' between endpoints show any PMTU problems? If you say "large payload" then anything over the MTU along the path will get DF set (even if some device along the path doesn't handle it correctly), right? You say "consistently". Do you have a (tcpdump/wireshark) fix on which fragment gets lost? Does the problem only occur with VPN data (as in IDS-like scrubbing)? If you use say 'nemesis' with UDP payload less than say 1300 (and have endpoints MTU 1300?) get frags assembled OK? And maybe a stupid question but since you're a paying customer, would your service provider be inclined to help?

We can't see any problems at all other than this one fragment missing. Apparently our partner reduced their MTU to 1400 (which we can't prove, and possibly doubt) and we saw no change. We can routinely see a missing fragment every minute, exactly the same one, and we have also confirmed that the fragment is leaving out peer correctly. But quite how this woudl ever get tracked down is beyond me. nemesis does look to be a good bet and will try and persue that though, a committee of managers are currently preventing any useful progress though!

Crap. The reason why I asked if you knew if it was a specific packet in sequence every time was because if things get fragmented some routers have problems (as only earlier packets will have the headers) linking up the frag and consequently refuse to send it on. Stuff getting lost en route doesn't look like something you'd comfortably troubleshoot yourself. Maybe prod your provider to open a ticket with their provider slash carrier? I spose there's no client in same ASN space on an approximate route where things work OK, and all your other client tunnels work OK, right? BTW, what VPN solution are we talking about hardware or software?


ROTFL. That's an original way of saying it's weekend...

Our carrier and the peers, which apparently have a direct interface on their respective AS's are being a little useful, more so than you'd expect based on the subtlety of the issue. We're on an ipsec VPN, which has been fine for a year. Now we have the very same ultra repeatable issue every minute...

And you know, it's not a weekend for us this weekend. Our partner (who we are massively closely connected to in terms of the whole point we both exist) shut up shop friday 6pm, despite them blatantly needing to stay involved. And the wife isn't happy with me.

May I also ask, what OS on your client side?

Well this is hardware firewalls, Juniper to PIX

Cisco with Juniper. They should like each other.
So, if it is really big problem and you or anyone really want to fix it quickly, I can suggest only one way.
You need to find another way to connect them. For example another ISP, or just bring one to other.
You need to exclude ISP.
If there was not changes in configuration on both of them, it can be ISP.

Oh they DO like each other, it's been fine for a year, and we know the traffic is going awol on the way. The irony here is that this was done as a vpn as it was cheaper than a leased circuit. Business impact means this is about 10000 times more expensive than a piddly kilostream circuit!

Other thing you can easily do is to connect two LINUX computer instead of this firewalls and ping each other to see if will be any data loss. It can give you idea about entire condition.

Have you checked if the "evil" bit is set?