LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (http://www.linuxquestions.org/questions/linux-software-2/)
-   -   ftp connections hanging forever in CLOSE, CLOSE_WAIT state (http://www.linuxquestions.org/questions/linux-software-2/ftp-connections-hanging-forever-in-close-close_wait-state-120140/)

mxyzplk 11-26-2003 09:54 AM

ftp connections hanging forever in CLOSE, CLOSE_WAIT state
 
Having a weird problem that just started recently. I'm running wu-ftpd 2.6.2(1) on a Red Hat linux Dell box (2.2.19 kernel). This has been working fine (well, reasonably fine) for a long time. Starting after its last reboot, ftp connections ceased to close properly and simply hang open forever.

We have a standard xinetd-spawns-wuftpd-process setup. The ftpd and xinetd idle timeouts are set and working, but something's going wrong when it tries to close its network connection. They get hung in CLOSE or CLOSE_WAIT states, and what's worse they seem to all be staying on the CPU - this jacks our load average up into the 200+ range in short order, where other services start to fail. We have to manually kill FTP processes to keep the box up.

ps listing for example hung FTP processes (hours past the idle timeout):
ftp 32765 1 3 08:23 ? 00:14:27 ftpd: 194.120.84.9: ftp: ABOR
ftp 300 1 3 08:23 ? 00:14:27 ftpd: 194.120.84.9: ftp: ABOR
ftp 1026 1 4 08:46 ? 00:13:55 ftpd: 194.120.84.9: ftp: ABOR

ftp 9228 8739 4 11:44 ? 00:07:04 ftpd: 141.37.154.230: ftp: IDLE
ftp 9382 8739 4 11:49 ? 00:06:36 ftpd: 141.37.154.230: ftp: IDLE
ftp 9390 8739 4 11:49 ? 00:06:34 ftpd: 141.37.154.230: ftp: IDLE
ftp 9402 8739 4 11:49 ? 00:06:33 ftpd: 141.37.154.230: ftp: IDLE
ftp 9458 8739 4 11:52 ? 00:06:22 ftpd: 141.37.154.230: ftp: IDLE

netstat listing for these IPs:
tcp 1 0 ftp.ni.com:ftp 194.120.84.9:16740 CLOSE_WAIT
tcp 1 0 ftp.ni.com:ftp 194.120.84.9:49262 CLOSE_WAIT
tcp 1 0 ftp.ni.com:ftp 194.120.84.9:48831 CLOSE_WAIT

tcp 0 99 ftp.ni.com:ftp 141.37.154.230:1814 CLOSE
tcp 0 99 ftp.ni.com:ftp 141.37.154.230:1789 CLOSE
tcp 0 99 ftp.ni.com:ftp 141.37.154.230:1787 CLOSE
tcp 0 99 ftp.ni.com:ftp 141.37.154.230:1786 CLOSE
tcp 0 99 ftp.ni.com:ftp 141.37.154.230:1753 CLOSE

A sample from top:
2:38pm up 4 days, 1:48, 6 users, load average: 54.07, 51.60, 49.06
211 processes: 154 sleeping, 57 running, 0 zombie, 0 stopped
CPU0 states: 89.1% user, 10.0% system, 88.0% nice, 0.0% idle
CPU1 states: 98.0% user, 1.0% system, 98.0% nice, 0.0% idle
Mem: 1048144K av, 973060K used, 75084K free, 0K shrd, 20976K buff
Swap: 0K av, 0K used, 0K free 792532K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
14096 root 18 0 1416 1416 964 R 10.6 0.1 0:00 top
24088 ftp 19 10 1268 1268 976 R N 7.9 0.1 24:37 in.ftpd
5741 ftp 19 10 1264 1264 976 R N 7.9 0.1 12:25 in.ftpd
5759 ftp 19 10 1264 1264 976 R N 7.9 0.1 4:48 in.ftpd
13648 ftp 20 10 1264 1264 976 R N 7.9 0.1 0:09 in.ftpd
28979 ftp 19 10 1264 1264 976 R N 7.1 0.1 19:18 in.ftpd
9448 ftp 19 10 1264 1264 976 R N 7.1 0.1 6:50 in.ftpd
10553 ftp 19 10 1264 1264 976 R N 7.1 0.1 5:36 in.ftpd
13649 ftp 19 10 1264 1264 976 R N 7.1 0.1 0:09 in.ftpd
9386 ftp 18 10 1264 1264 976 R N 6.2 0.1 7:00 in.ftpd
10555 ftp 19 10 1264 1264 976 R N 6.2 0.1 5:36 in.ftpd
11071 ftp 18 10 1264 1264 976 R N 6.2 0.1 3:10 in.ftpd
13641 ftp 19 10 1264 1264 976 R N 6.2 0.1 0:09 in.ftpd

Does anyone have any idea what might be causing this? We haven't been able to correlate to any changes made around that time to network/firewall/server/etc. I've rebooted again and fiddled with ftpd config, xinetd config, and DNS config to no avail.

Thanks,
Ernest


All times are GMT -5. The time now is 10:23 AM.