Incoming external HTTP connection not getting ESTABLISHED (stuck in SYN_RECV)
 

Hello,

I've come across an issue I just can't seem to resolve. I'm setting up a test web app which I can successfully connect to on my LAN (192.168.1.*) but cannot connect to from the outside world. Upon using "netstat -t" I've found the connection is "stuck" in SYN_RECV state:

tcp 0 0 [my_web_host]:http [exteranl_pc]:30473 SYN_RECV


Looking at "tcpdump -i eth2 tcp port 80":

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
21:17:50.625575 IP [external_pc].30473 > [my_web_host].http: Flags [S], seq 4169122453, win 65535, options [mss 1460,nop,wscale 0,sackOK,TS val 1929944507 ecr 0], length 0
21:17:50.625646 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
21:17:53.620386 IP [external_pc].30473 > [my_web_host].http: Flags [S], seq 4169122453, win 65535, options [mss 1460,nop,wscale 0,sackOK,TS val 1929947507 ecr 0], length 0
21:17:53.620437 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
21:17:54.824797 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
21:17:56.821979 IP [external_pc].30473 > [my_web_host].http: Flags [S], seq 4169122453, win 65535, options [mss 1460,nop,wscale 0,sackOK,TS val 1929950707 ecr 0], length 0
21:17:56.822029 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
21:18:00.021567 IP [external_pc].30473 > [my_web_host].http: Flags [S], seq 4169122453, win 65535, options [mss 1460,sackOK,eol], length 0
21:18:00.021620 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
21:18:00.824802 IP [my_web_host].http > [external_pc].30473: Flags [S.], seq 1535877877, ack 4169122454, win 14600, options [mss 1460,nop,nop,sackOK], length 0
...
...

I'm pretty sure my firewall is ok, but here's the "iptables -L" output:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- 192.168.1.0/24 anywhere state NEW tcp dpt:ssh
ACCEPT tcp -- 192.168.1.0/24 anywhere state NEW tcp dpt:postgres
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

My Netgear WNDR3400 router, I have port 80 set to forward to my_web_host. I've accomplished this same thing awhile back, albeit with different equipment. It appears to me that I have my Linux server set up correctly, and the tcpdump seems to indicate to me that it is responding the the SYN request with a SYN_ACK, hence the SYN_RECV state. It leaves me to think the router might be the problem but I have no idea how to inspect the tcp packets there. Or is it just something I've overlooked on my Linux server?

FYI, I came across a post that seemed similar and suggested setting tcp_window_scaling=0 and tcp_timestamps=0. I've tried this and it still doesn't work.

Any help into this issue would be greatly appreciated. I'd be glad to supply more details if necessary.

Many thanks,
tobias

can you ping the remote pc from the server? Clearly the server is responding to the client, but looks like the client never receives the response. The response should be part of a stateful flow, so the fact that the original SYN got in says that the response should get back, as long as the routing you have there isn't weird and asymmetric. How many clients in how many locations have you tried? do you know the fault is not at their end? can you do a tcpdump on the client and compare? If you add an "-e" to the tcpdump you'll see the MAC addresses in the traffic, to ensure that the local device that send you the packet (i.e. the router) is the device that is sent to reply.

Thanks for the response. I've tried from 2 distinct clients, both fail to establish and timeout. I hadn't tried pinging from my server, but interestingly this is what I get:


PING [external_pc.com] ([external_pc_ip]) 56(84) bytes of data.
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=1 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=2 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=3 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=4 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=5 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=6 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=7 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=10 Packet filtered
From 41332a3a.cst.lightpath.net (65.51.42.58) icmp_seq=11 Packet filtered
...
...

Pinging www.yahoo.com works just fine:
PING any-fp3-real.wa1.b.yahoo.com (98.139.183.24) 56(84) bytes of data.
64 bytes from ir2.fp.vip.bf1.yahoo.com (98.139.183.24): icmp_seq=1 ttl=52 time=31.0 ms
64 bytes from ir2.fp.vip.bf1.yahoo.com (98.139.183.24): icmp_seq=2 ttl=52 time=66.4 ms
...
...

Unfortunately I don't have access to investigate from either of these locations at this time. I'm going to look into these "Packet filtered" ping issues to see if I can get anywhere and see if the "-e" tcpdump option will get me anywayre. But of course any further insight is much appreciated!

Kind Regards,
tobias

Turns out Cablevision (my provider) "blocks" port forwarding on their routers, so they say. I am of course using theirs at the moment (the Netgear WNDR3400). Their reasoning was so as not to confuse the grandmas out there. Of course I got confused as a result - I was able to go onto the router and seemingly configure it to forward HTTP w/o any sort of notification that it wouldn't work.

Anyway, they say I can simply use my own router instead for port forwarding. I'll be getting one shortly - any advice on a decent one for relatively simple home networking (including port forwarding obviously :)? A couple linux boxes, a windows pc, several wireless devices, including iPad2 and Wii. This will sit behind the provider's modem - an Arris TM802G.

I'll be sure to post here the results after I purchase the router to verify all i needed was my own router.

Regards,
tobias

I'm sure most anything you'll get will be fine, just don't get an ADSL router, ethernet only. The "classic" would be a netgear WRT54G though, which has been a hackers play thing for a long time, and spawned DDWRT / OpenWRT embeded operating systems you might like to play with (potentially on other bits of kit as it works on lots of devices these days.).

don't quite follow what your ISP says, they have modded the router firmware?? you'd think you'd just remove the menu option, not what... break inbound connection tracking??

This post helped me!
 

Hi,

I just registered to get this posted. Just updated my server hardware and got my old Arch Linux installation running again. Everything worked fine for local connections, but however external request to http, ssh and anything else (except icmp) did not work. I was fiddling around with this problem for two days since now, reading the following line.
So i did this and got the following:
When you look at the destination mac in the third line, when hostname is responding to the SYNACK, you can see the problem. The response is redirected to 00:0c:6e:0e:4a:90 instead of 00:01:2e:37:58:ba . I then was searching for the wrong MAC ending on 90. It was still stored inside the router (don't know which table). After removing the deactivated DHCP entry and restarting the router, the packets are now routed correctly.

Hoping this post helps anybody, who has the same problem... Thank you for your idea acid_kewpie.

Greets!