I am using squid as a cache server for a webstite.
So basically when some one hits my web page squid serves it from cache and if it can't the web server replies to the request.
My problem is that when I have a problem too many TCP connections some of them get dropped. I have all ready tweaked some of my kernel parameters but the problem remains.
the output of netstat -s shows quite a few problems with tcp but I can not figure out why this is happening . I will display the info I get from netstat -s. The last paragraph looks scary.
Ip:
324538214 total packets received
155 with invalid addresses
0 forwarded
0 incoming packets discarded
324522743 incoming packets delivered
311408578 requests sent out
635 outgoing packets dropped
31 dropped because of missing route
188 reassemblies required
93 packets reassembled ok
Icmp:
18522 ICMP messages received
7 input ICMP message failed.
ICMP input histogram:
destination unreachable: 2145
timeout in transit: 299
source quenches: 1
echo requests: 16070
16077 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 7
echo replies: 16070
Tcp:
4575069 active connections openings
33767002 passive connection openings
133019 failed connection attempts
173050 connection resets received
972 connections established
324461010 segments received
297923659 segments send out
13418423 segments retransmited
5791 bad segments received.
387427 resets sent
Udp:
41701 packets received
4 packets to unknown port received.
0 packet receive errors
41713 packets sent
TcpExt:
338833 SYN cookies sent
413142 SYN cookies received
354155 invalid SYN cookies received
132901 resets received for embryonic SYN_RECV sockets
2532 packets pruned from receive queue because of socket buffer overrun
117 ICMP packets dropped because they were out-of-window
32503602 TCP sockets finished time wait in fast timer
1676 packets rejects in established connections because of timestamp
1162164 delayed acks sent
353 delayed acks further delayed because of locked socket
Quick ack mode was activated 29771 times
2846425 times the listen queue of a socket overflowed
2846425 SYNs to LISTEN sockets ignored
23405 packets directly queued to recvmsg prequeue.
297 of bytes directly received from prequeue
18032547 packet headers predicted
2 packets header predicted and directly queued to user
131489399 acknowledgments not containing data received
42919388 predicted acknowledgments
757003 times recovered from packet loss due to fast retransmit
2826050 times recovered from packet loss due to SACK data
108 bad SACKs received
Detected reordering 2282 times using FACK
Detected reordering 1679 times using SACK
Detected reordering 4064 times using reno fast retransmit
Detected reordering 4268 times using time stamp
1827 congestion windows fully recovered
8512 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 6175
36408 congestion windows recovered after partial ack
2872673 TCP data loss events
TCPLostRetransmit: 2653
340515 timeouts after reno fast retransmit
1183718 timeouts after SACK recovery
408401 timeouts in loss state
4482533 fast retransmits
282823 forward retransmits
3182137 retransmits in slow start
1433259 other TCP timeouts
TCPRenoRecoveryFail: 482745
742756 sack retransmits failed
52453 packets collapsed in receive queue due to low socket buffer
23694 DSACKs sent for old packets
37 DSACKs sent for out of order packets
119706 DSACKs received
13905 DSACKs for out of order packets received
7611 connections reset due to unexpected data
2571 connections reset due to early user close
39572 connections aborted due to timeout
1 times unabled to send RST due to no memory
As well I tried from the same host that squid runs.
ab -c 1000 -n 30000
www.myexample.com
and the results were
oncurrency Level: 1000
Time taken for tests: 28.668550 seconds
Complete requests: 30000
Failed requests: 27556
(Connect: 0, Length: 27556, Exceptions: 0)
Write errors: 0
Total transferred: 2023484423 bytes
HTML transferred: 2012078837 bytes
Requests per second: 1046.44 [#/sec] (mean)
Time per request: 955.618 [ms] (mean)
Time per request: 0.956 [ms] (mean, across all concurrent requests)
Transfer rate: 68927.76 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 6.8 0 64
Processing: 254 940 176.6 957 6403
Waiting: 22 775 167.5 809 2652
Total: 254 941 175.6 957 6460
Percentage of the requests served within a certain time (ms)
50% 957
66% 973
75% 988
80% 1003
90% 1029
95% 1070
98% 1117
99% 1125
100% 6460 (longest request)
If you have any idea why this is happenning or what can I do to accommodate more tcp connections, I would be gratefull.
Please help. It's a sports website and on Wensday is champions league day and it's going to be a massacre again.
