Hi guys,

I'm having some trouble tracking down an *outbound* connection problem on my Linode / Debian Unstable setup. Hopefully someone can point me in the right direction. Here's the situation:

I have a Nodejs script running that polls a couple external machines/routers ever minute. It sets the idle socket timeouts long enough where it can reuse existing connections on subsequent polls. Sometimes the other end closes the sockets, but the script reconnects if it needs to. Everything works great for several days.

Sometimes a machine quits responding in a timely manner (> 5 seconds) and I have node abort the connection attempt for that poll cycle calling http://nodejs.org/api/net.html#net_socket_destroy. This works fine for occasional bad connections, but in certain cases (perhaps multiple bad consecutive attempts) something in Debian's network stack actually screws over and starts refusing outbound connections, consistently resulting in ECONNREFUSED. I tried to figure out what could be going on but have been unsuccessful.

I tried telneting to the remote machines from the Linode SSH, Connection refused. Telnet to google:80 works okay. The firewall log on the target router shows the connection packet was received and accepted. I can telnet to the machine itself locally just fine from Windows. Linode/Debian still says conn refused. I looked at netstat -ton output and did not see an unusually large number of CLOSE_WAIT or other hanging sockets.

BUT...restarting the Linode fixes everything.

I tried restarting just the networking service but it kicks me off ssh, so i cannot tell if that fixes anything, but it doesnt let me ssh back in until i reboot the machine through the Linode manager.

if anyone has some ideas, please let me know.

thanks!

- Strace to NodeJS process and run tcpdump so you can actually see what's happening?
- Do research on network-related sysctls and see if tuning settings may help?
- Determine what kernel version the Linode runs and try another version?
- Run a debug kernel and see if it spills any clues?

thanks for the advice. i will keep posting more info as i find it. i did 2 tcpdump runs, one when everything was working (after restart yesterday) and one when it was broken (this morning) via:

tcpdump -vvv -i eth0 tcp port not 22 and port not 443 -w dump_bad.pcap

the port exclusion is just to get rid of some noise.

the capture done during the brokenness shows no request packets going out. none. again, i made sure i could connect to the device from another machine, this time telnet from a different Linode. everything connected instantly.

EDIT: never mind, for some reason the nodejs script/process died.

i would like to see what happens when i try to telnet outbound, but when i run tcpdump over ssh, it takes over my interface and i cannot do anything until i stop tcpdump. is there a way to run it in the background so i can do some logging of telnet connection attempts?

Start tcpdump from an 'at' command or run tcpdump and add an ampersand or run tcpdump inside screen in a separate window?..