openvpn strange problem!

chakoshi · 10-23-2010, 06:36 AM

Hi, it's a few days I'm trying hard to run an openvpn server on a openvz VPS, the problem is packets sent from openvpn server process, doesnt reach the client, so connection is never stablished (I run tcpdump on server and wireshark on client to carefully investigate whats wrong), the first guess is that a kind of firewall is blocking traffic (I tried connecting to server through different ISP's but it's possible the national network provider applied some filtering but it cant be on IP,src port or dst port as I'd tried different configuration. what about deep packet inspection technics, is it possible to block my traffic?) but at exactly the same time I can transmit UDP packets using netcat from server to the guest. there is no firewall enabled in between, I had tried, tcp and udp, tried both open vpn and openvpnAS and tried any thing one can imagine! I really need some suggestions!

the VPN is configured as a routed (TUN) type on debian

Noway2 · 10-23-2010, 08:54 AM

OpenVPN uses encrypted traffic. It is highly unlikely that an ISP is able to distinguish this type of data and deliberately block it. Even if they could, unless running a vpn is prohibited by the TOS (some ISPs consider it to be a "business" function, it would be a strange thing to block. Consequently, you are most likley looking at a more mundane problem.

Are you able to telnet between the host and client on the port that you have set up for openVPN? If not, try a different port and run open VPN on that, at least as a test.

Failing that, what error messages are you getting, if any. What do your logs indicate? Are the openVPN processes running and did they start with any errors?

chakoshi · 10-23-2010, 10:42 AM

Quote:

Originally Posted by Noway2

OpenVPN uses encrypted traffic. It is highly unlikely that an ISP is able to distinguish this type of data and deliberately block it. Even if they could, unless running a vpn is prohibited by the TOS (some ISPs consider it to be a "business" function, it would be a strange thing to block. Consequently, you are most likley looking at a more mundane problem.

Are you able to telnet between the host and client on the port that you have set up for openVPN? If not, try a different port and run open VPN on that, at least as a test.

Failing that, what error messages are you getting, if any. What do your logs indicate? Are the openVPN processes running and did they start with any errors?

thank you for the reply. from the client I cant telnet the port and says "could not open port" but when I try to connect the vpn client as I dump the network traffic and monitor it in realtime on both client and server, the client initiate the connection, openvpn get the request and the server give these messages:

Code:

Initialization Sequence Completed
MULTI: multi_create_instance called
myIP:62206 Re-using SSL/TLS context
myIP:62206  LZO compression initialized
myIP:62206  Control Channel MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
myIP:62206  Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
myIP:62206  Local Options hash (VER=V4): '14168603'
myIP:62206  Expected Remote Options hash (VER=V4): '504e774e'
myIP:62206  TLS: Initial packet from 94.183.127.131:62206, sid=08991883 6ffae1ac

from this stage it is seen in network dumps that client sending udp packets, server get the packets then sends the responses (seen on the server network interface) but these packages are not get to the client , the odd thing is that at the same time if I make another ssh session to server and send UDP packets to client usig netcat,these netcat generated packages reaches the client easily.

then affter a minute the server prompt:

Code:

TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
myIP:58076 TLS Error: TLS handshake failed
myIP:58076 SIGUSR1[soft,tls-error] received, client-instance restarting

ixsys · 10-26-2010, 09:47 AM

hi chakoshi,

i have exactly the same problem with two debian 5.x boxes talking openvpn to each other:

* the client initiates the UDP connection originating from an unprivileged port (>35000) addressing the server at port 1194

* the server responds the request using the established connection

* client and server exchange alltogether 10 packets the the server starts to send packets which never reach the client

it's indeed bizare since i have about 40 working openvpn installations out in the wild, same os, same setup etc.; it's the first time i encounter a problem like this...

when i change the setup to tcp everything works as expected, which means that authentication (certificates, tls key, tls-auth etc.) is ok;

when i reproduce the network communication with netcat on both sides, everything works fine, which means firewall setup is ok as well;

i have to stress that the setup was working without any problems for a long time (> 1 year); the trouble appeared without any changes of configuration;

did you find out anything? i'm still puzzled....

Noway2 · 10-27-2010, 04:26 AM

Given what is and isn't working, I can understand why you would be curious about an ISP blocking openVPN traffic.

The part about TCP working where UDP didn't suggests that you may have an underlying network connection, or quality of service, issue. TCP has more built in redundancy due to its connection nature. With TCP dropped packets will get re-transmitted and the end points will make a best effort to ensure that all of the data has been transmitted and received. With UDP, this responsibility falls to the application and appears to be failing.

chakoshi · 10-27-2010, 07:25 AM

Quote:

Originally Posted by ixsys

hi chakoshi,

i have exactly the same problem with two debian 5.x boxes talking openvpn to each other:

* the client initiates the UDP connection originating from an unprivileged port (>35000) addressing the server at port 1194

* the server responds the request using the established connection

* client and server exchange alltogether 10 packets the the server starts to send packets which never reach the client

it's indeed bizare since i have about 40 working openvpn installations out in the wild, same os, same setup etc.; it's the first time i encounter a problem like this...

when i change the setup to tcp everything works as expected, which means that authentication (certificates, tls key, tls-auth etc.) is ok;

when i reproduce the network communication with netcat on both sides, everything works fine, which means firewall setup is ok as well;

i have to stress that the setup was working without any problems for a long time (> 1 year); the trouble appeared without any changes of configuration;

did you find out anything? i'm still puzzled....

very similar to my problem, but I have the problem also in TCP, in TCP client receives response but it seems the handshake never completes! I'm working on it more, if any results found I'll post it.I've posted more descriptions and logs at https://forums.openvpn.net/viewtopic.php?f=4&t=7209

Quote:

Given what is and isn't working, I can understand why you would be curious about an ISP blocking openVPN traffic.

yes I'm thinking about advanced filtering technics(signature based,...)

ixsys · 10-27-2010, 06:09 PM

Quote:

The part about TCP working where UDP didn't suggests that you may have an underlying network connection, or quality of service, issue. TCP has more built in redundancy due to its connection nature. With TCP dropped packets will get re-transmitted and the end points will make a best effort to ensure that all of the data has been transmitted and received. With UDP, this responsibility falls to the application and appears to be failing.

i totally agree, and indeed the vpn tunnel is intended to connect sites in germany and dubai, which implies a low quality link within the AE;
BUT i would expect a link of poor quality to produce different glitches, e.g. low transfer speed, sporadic connection loss etc.

the fact is, i can reproduce the problem as described every time; thus i have to assume some kind of QoS or filter, BUT (again) at the same time there's a perfectly running IPSEC tunnel on the same host which utilizes UDP as well;

by the way, i discovered a major error in my former description of the problem:

after the initial conversation the client is sending a UDP packet to the server which never reaches its destination; after not getting a response to this packet from the server the client sends another 2 packets after very short waiting time, see the tcpdump trace:

Client (85.183.x.x)

Code:

00:57:53.825505 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 42
00:57:53.987098 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 54
00:57:53.987806 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 50
00:57:53.988165 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 135
00:57:54.526387 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 154
00:57:54.526943 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:54.527197 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:54.527921 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 50
00:57:54.528290 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 50
00:57:54.528626 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 50

Server (94.200.x.x)

Code:

00:57:53.906056 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 42
00:57:53.906056 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 54
00:57:54.066043 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 50
00:57:54.070043 IP 85.183.x.x.41153 > 94.200.x.x.1194: UDP, length 135
00:57:54.442014 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 154
00:57:54.446014 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:54.446014 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:54.446014 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:56.693840 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:56.693840 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:56.693840 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:56.693840 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:58.941665 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:58.941665 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:58.941665 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:57:58.941665 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:58:00.065578 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:58:00.065578 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:58:00.065578 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142
00:58:00.065578 IP 94.200.x.x.1194 > 85.183.x.x.41153: UDP, length 142

any idea is appreciated...

chakoshi · 10-28-2010, 02:38 AM

Quote:

i totally agree, and indeed the vpn tunnel is intended to connect sites in germany and dubai, which implies a low quality link within the AE;
BUT i would expect a link of poor quality to produce different glitches, e.g. low transfer speed, sporadic connection loss etc.

the fact is, i can reproduce the problem as described every time; thus i have to assume some kind of QoS or filter, BUT (again) at the same time there's a perfectly running IPSEC tunnel on the same host which utilizes UDP as well;

by the way, i discovered a major error in my former description of the problem:

after the initial conversation the client is sending a UDP packet to the server which never reaches its destination; after not getting a response to this packet from the server the client sends another 2 packets after very short waiting time, see the tcpdump trace:

exactly the strange thing is that the problem is with openvpn UDP packets, I'm not sure what kind of characteristics those packets have! I tried to monitor both sides for several minuts, I see that about 98% of the packets sent by openvpn to client drops and only two percents seen on client (I kept running a ping between TUN devices of client and server then let openvpn sides try to connect), the ping response is almost "request timeout" and some times it succedes to ping.
just like you I think its a kind of filtering using QoS methods, as it's not the first time in my region.
how do you tried the IPSEC tunnel?
also I see the port 1194 in your log, in my region packets with src or dest address of 1194 are filltered. try another port.

Noway2 · 10-29-2010, 04:24 AM

I don't know how to answer your question about ipsec, but if your problem is a deliberate filtering of udp on port 1194, I can think of three other potential workarounds: 1 - use a different port, 2 - switch to TCP and / or a different port, 3 - use a different program for your VPN, such as SSH.

chakoshi · 10-29-2010, 04:55 AM

Quote:

Originally Posted by Noway2

I don't know how to answer your question about ipsec, but if your problem is a deliberate filtering of udp on port 1194, I can think of three other potential workarounds: 1 - use a different port, 2 - switch to TCP and / or a different port, 3 - use a different program for your VPN, such as SSH.

of course, at the first step I changed the ports, but the other problems arise, now I can only use openvpn in udp mode and with satitic key, tcp mode or using ca and cert files faile to handshake and initialize connection (may be because of heavy packets loss). unfortunately the only vpn methods I can use in openvz other than openvpn is ssh, currently I'm routing my traffic through ssh tunnel, until I find a solution for openvpn problem.