listen queue of a socket overflowed, process is not taking new requests

bhaskarna · 06-23-2006, 05:33 AM

Hi,
We are running tomcat server on RHEL 3 (Linux <myhostname> 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT 2005 i686 i686 i386 GNU/Linux).
After the loadtests on port 4080, we noticed that the tomcat server stopped taking new requests in port 4080. But it was still continued to take requests in 443.

In netstat -s, we noticed:

$ netstat -s|grep overflow
421277 times the listen queue of a socket overflowed
421277 SYNs to LISTEN sockets ignored

even after 10hrs after stopping loadtests, we were not able to send new requests to port 4080. When we checked netstat -s, we noticed that the request was in SYN_RECV for long.

I took the proc status of both the java processes.

What does this hung mean? why does process stopped accepting new requests? Are there any OS limitations/restrictions that freezes a particular port after n number of requests?

what should I do to make my server stable?

During hung time, i took the proc status. This is how it look like:

$ cat /proc/23814/status
Name: jsvc
State: S (sleeping)
SleepAVG: 88%
Tgid: 23814
Pid: 23814
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 32
Groups: 0 1 2 3 4 5 6 9 10 20 31
VmSize: 2840 kB
VmLck: 0 kB
VmRSS: 616 kB
VmData: 168 kB
VmStk: 1240 kB
VmExe: 25 kB
VmLib: 1355 kB
StaBrk: 08050000 kB
Brk: 087fb000 kB
StaStk: bfeccaa0 kB
ExecLim: 0804f000
Threads: 1
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180004003
CapInh: 0000000000000000
CapPrm: 00000000fffffeff
CapEff: 00000000fffffeff

$ cat /proc/23814/status
Name: jsvc
State: S (sleeping)
SleepAVG: 88%
Tgid: 23815
Pid: 23815
PPid: 23814
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 100 100 100 100
FDSize: 256
Groups: 10 100
VmSize: 330824 kB
VmLck: 0 kB
VmRSS: 221036 kB
VmData: 260784 kB
VmStk: 1272 kB
VmExe: 25 kB
VmLib: 206695 kB
StaBrk: 08050000 kB
Brk: 0962b000 kB
StaStk: bfeccaa0 kB
ExecLim: bfecd000
Threads: 53
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 1000000180015ccf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

sundialsvcs · 07-02-2006, 11:26 AM

No. I can only conclude that the process must be frozen in some kind of endless-loop where it is no longer listening.

bhaskarna · 07-18-2006, 03:06 AM

I see this happening even when the process is able to listen to the request. So, i don't think overflow is happening when process gets into hung state. Looks like there is someother issue in this box.