Hi,
We are getting network issue for NFS and memcached in our network. But getting difficulties to diagnose both.
We are not getting any logs in dmesg and messages. Please refer the netstat and ifconfig outputs as follow.
Please let us know if we need to change any kernel settings. We have upgraded network drivers but issue still there.
Operating system- CentOS release 5.4 (Final)
Kernel - 2.6.18-274
====================================================
ifconfig
Code:
RX packets:97198166 errors:3104 dropped:11 overruns:0 frame:3104
TX packets:27200159 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX packets:110829843 errors:5 dropped:14 overruns:0 frame:5
TX packets:132306261 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
=========================================
[root@host1]# netstat -s
Code:
Ip:
54308864 total packets received
688 with invalid addresses
0 forwarded
0 incoming packets discarded
53901555 incoming packets delivered
64669990 requests sent out
6 dropped because of missing route
Icmp:
9997 ICMP messages received
3 input ICMP message failed.
ICMP input histogram:
destination unreachable: 163
echo requests: 9828
echo replies: 6
10554 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 720
echo request: 6
echo replies: 9828
IcmpMsg:
InType0: 6
InType3: 163
InType8: 9828
OutType0: 9828
OutType3: 720
OutType8: 6
Tcp:
6581507 active connections openings
1607052 passive connection openings
3526 failed connection attempts
15216 connection resets received
2 connections established
52338714 segments received
62884099 segments send out
10711 segments retransmited
0 bad segments received.
6295 resets sent
Udp:
1550750 packets received
698 packets to unknown port received.
0 packet receive errors
1764626 packets sent
TcpExt:
169 resets received for embryonic SYN_RECV sockets
188 TCP sockets finished time wait in fast timer
28 time wait sockets recycled by time stamp
7987454 TCP sockets finished time wait in slow timer
1 packets rejects in established connections because of timestamp
471256 delayed acks sent
1277 delayed acks further delayed because of locked socket
Quick ack mode was activated 2550 times
1164815 packets directly queued to recvmsg prequeue.
2419208 packets directly received from backlog
1339602178 packets directly received from prequeue
16421204 packets header predicted
411964 packets header predicted and directly queued to user
12740664 acknowledgments not containing data received
7724081 predicted acknowledgments
341 times recovered from packet loss due to SACK data
Detected reordering 1 times using FACK
Detected reordering 5 times using SACK
Detected reordering 2 times using time stamp
4 congestion windows fully recovered
53 congestion windows partially recovered using Hoe heuristic
756 congestion windows recovered after partial ack
1186 TCP data loss events
TCPLostRetransmit: 2
643 timeouts after SACK recovery
98 timeouts in loss state
1091 fast retransmits
108 forward retransmits
1719 retransmits in slow start
6094 other TCP timeouts
87 sack retransmits failed
2 times receiver scheduled too late for direct processing
2550 DSACKs sent for old packets
496 DSACKs received
365 connections reset due to unexpected data
213 connections aborted due to timeout
IpExt:
InBcastPkts: 1396
++++++++++++++++++++++++++
Proto Recv-Q Send-Q State PID/Program name
tcp 0 4365 ESTABLISHED 11352/httpd
tcp 0 4365 ESTABLISHED 19753/httpd
tcp 0 4365 ESTABLISHED -
tcp 0 4374 FIN_WAIT1 14063/httpd
tcp 0 4435 ESTABLISHED 6652/httpd
tcp 0 4505 FIN_WAIT1 3232/httpd
tcp 0 4531 FIN_WAIT1 1917/httpd
tcp 0 4532 FIN_WAIT1 15552/httpd
tcp 0 4611 FIN_WAIT1 16370/httpd
tcp 0 4788 ESTABLISHED 5170/httpd
tcp 0 4818 FIN_WAIT1 15557/httpd
tcp 0 5026 ESTABLISHED 5494/httpd
tcp 0 5130 ESTABLISHED -
tcp 0 5497 FIN_WAIT1 12466/httpd
tcp 0 5645 ESTABLISHED 22611/httpd
tcp 0 5773 ESTABLISHED 28445/httpd
tcp 0 5841 ESTABLISHED 1653/httpd
tcp 0 6010 ESTABLISHED 22245/httpd
tcp 0 6228 ESTABLISHED 23534/httpd
tcp 0 6392 FIN_WAIT1 24441/httpd
tcp 0 6429 ESTABLISHED 1917/httpd
tcp 0 6457 FIN_WAIT1 23745/httpd
tcp 0 6922 FIN_WAIT1 6649/httpd
tcp 0 6988 FIN_WAIT1 14158/httpd
tcp 0 7256 ESTABLISHED 29844/httpd
tcp 0 7560 FIN_WAIT1 26647/httpd
tcp 0 7619 FIN_WAIT1 12465/httpd
tcp 0 7742 ESTABLISHED 26388/httpd
tcp 0 7922 ESTABLISHED 14985/httpd
tcp 0 7922 ESTABLISHED 14158/httpd
tcp 0 8529 ESTABLISHED 12462/httpd
tcp 0 8569 FIN_WAIT1 23170/httpd
tcp 0 8630 ESTABLISHED 20565/httpd
tcp 0 8694 ESTABLISHED 27032/httpd
tcp 0 8705 ESTABLISHED 2373/httpd
tcp 0 8772 ESTABLISHED -
tcp 0 8779 ESTABLISHED 6649/httpd
tcp 0 8902 ESTABLISHED 14334/httpd
tcp 0 9028 FIN_WAIT1 12118/httpd
tcp 0 9183 FIN_WAIT1 14985/httpd
tcp 0 9982 FIN_WAIT1 16363/httpd
tcp 0 10566 FIN_WAIT1 25911/httpd
tcp 0 16432 ESTABLISHED 31379/sshd: prod [p
tcp 0 16432 ESTABLISHED 22215/sshd: prod [p
tcp 0 25125 ESTABLISHED 30203/bash
tcp 0 25346 FIN_WAIT1 -
========================================================
Present Kernel settings-
Code:
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syn_retries = 3
net.ipv4.conf.all.log_martians = 1
net.ipv4.inet_peer_gc_mintime = 5
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_dsack = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 20
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_rfc1337 = 1
-thanks in Adv
