LinuxQuestions.org - TCP Socket Buffer Issues

I am using a 2-port LACP 10Gb NIC on an NFS server. OS is CentOS 5.4 with kernel 2.6.18-164.15.1.qtm.4.

Seeing lots of issues on 10Gb (netstat output below)

Looking at the net settings I see some that intuitively don't make a lot of sense when compared to eachother:

net.ipv4.tcp_rmem = 4194304 4194304 4194304
net.ipv4.tcp_wmem = 4194304 4194304 4194304
net.ipv4.tcp_mem = 196608 262144 393216
net.core.wmem_default = 129024
net.core.wmem_max = 1048575
net.core.optmem_max = 20480
net.core.rmem_default = 129024
net.core.rmem_max = 1048575

Anyone have any suggestions? I have 8 10Gb clients constantly reading & writing to this NFS server, posting full net. kernel settings & netstat from server below:

KERNEL:
net.unix.max_dgram_qlen = 1024
net.token-ring.rif_timeout = 600000
net.ipv4.ip_conntrack_max = 65536
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 1
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_buckets = 8192
net.ipv4.netfilter.ip_conntrack_count = 924
net.ipv4.netfilter.ip_conntrack_max = 65536
net.ipv4.conf.eth3.promote_secondaries = 0
net.ipv4.conf.eth3.force_igmp_version = 0
net.ipv4.conf.eth3.disable_policy = 0
net.ipv4.conf.eth3.disable_xfrm = 0
net.ipv4.conf.eth3.arp_accept = 0
net.ipv4.conf.eth3.arp_ignore = 0
net.ipv4.conf.eth3.arp_announce = 0
net.ipv4.conf.eth3.arp_filter = 0
net.ipv4.conf.eth3.tag = 0
net.ipv4.conf.eth3.log_martians = 0
net.ipv4.conf.eth3.bootp_relay = 0
net.ipv4.conf.eth3.medium_id = 0
net.ipv4.conf.eth3.proxy_arp = 0
net.ipv4.conf.eth3.accept_source_route = 0
net.ipv4.conf.eth3.send_redirects = 1
net.ipv4.conf.eth3.rp_filter = 1
net.ipv4.conf.eth3.shared_media = 1
net.ipv4.conf.eth3.secure_redirects = 1
net.ipv4.conf.eth3.accept_redirects = 1
net.ipv4.conf.eth3.mc_forwarding = 0
net.ipv4.conf.eth3.forwarding = 0
net.ipv4.conf.bond1.promote_secondaries = 0
net.ipv4.conf.bond1.force_igmp_version = 0
net.ipv4.conf.bond1.disable_policy = 0
net.ipv4.conf.bond1.disable_xfrm = 0
net.ipv4.conf.bond1.arp_accept = 0
net.ipv4.conf.bond1.arp_ignore = 0
net.ipv4.conf.bond1.arp_announce = 0
net.ipv4.conf.bond1.arp_filter = 1
net.ipv4.conf.bond1.tag = 0
net.ipv4.conf.bond1.log_martians = 0
net.ipv4.conf.bond1.bootp_relay = 0
net.ipv4.conf.bond1.medium_id = 0
net.ipv4.conf.bond1.proxy_arp = 0
net.ipv4.conf.bond1.accept_source_route = 0
net.ipv4.conf.bond1.send_redirects = 1
net.ipv4.conf.bond1.rp_filter = 1
net.ipv4.conf.bond1.shared_media = 1
net.ipv4.conf.bond1.secure_redirects = 1
net.ipv4.conf.bond1.accept_redirects = 1
net.ipv4.conf.bond1.mc_forwarding = 0
net.ipv4.conf.bond1.forwarding = 0
net.ipv4.conf.bond0.promote_secondaries = 0
net.ipv4.conf.bond0.force_igmp_version = 0
net.ipv4.conf.bond0.disable_policy = 0
net.ipv4.conf.bond0.disable_xfrm = 0
net.ipv4.conf.bond0.arp_accept = 0
net.ipv4.conf.bond0.arp_ignore = 0
net.ipv4.conf.bond0.arp_announce = 0
net.ipv4.conf.bond0.arp_filter = 1
net.ipv4.conf.bond0.tag = 0
net.ipv4.conf.bond0.log_martians = 0
net.ipv4.conf.bond0.bootp_relay = 0
net.ipv4.conf.bond0.medium_id = 0
net.ipv4.conf.bond0.proxy_arp = 0
net.ipv4.conf.bond0.accept_source_route = 0
net.ipv4.conf.bond0.send_redirects = 1
net.ipv4.conf.bond0.rp_filter = 1
net.ipv4.conf.bond0.shared_media = 1
net.ipv4.conf.bond0.secure_redirects = 1
net.ipv4.conf.bond0.accept_redirects = 1
net.ipv4.conf.bond0.mc_forwarding = 0
net.ipv4.conf.bond0.forwarding = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.tag = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.neigh.eth3.base_reachable_time_ms = 30000
net.ipv4.neigh.eth3.retrans_time_ms = 1000
net.ipv4.neigh.eth3.locktime = 99
net.ipv4.neigh.eth3.proxy_delay = 79
net.ipv4.neigh.eth3.anycast_delay = 99
net.ipv4.neigh.eth3.proxy_qlen = 64
net.ipv4.neigh.eth3.unres_qlen = 3
net.ipv4.neigh.eth3.gc_stale_time = 60
net.ipv4.neigh.eth3.delay_first_probe_time = 5
net.ipv4.neigh.eth3.base_reachable_time = 30
net.ipv4.neigh.eth3.retrans_time = 99
net.ipv4.neigh.eth3.app_solicit = 0
net.ipv4.neigh.eth3.ucast_solicit = 3
net.ipv4.neigh.eth3.mcast_solicit = 3
net.ipv4.neigh.bond1.base_reachable_time_ms = 30000
net.ipv4.neigh.bond1.retrans_time_ms = 1000
net.ipv4.neigh.bond1.locktime = 99
net.ipv4.neigh.bond1.proxy_delay = 79
net.ipv4.neigh.bond1.anycast_delay = 99
net.ipv4.neigh.bond1.proxy_qlen = 64
net.ipv4.neigh.bond1.unres_qlen = 3
net.ipv4.neigh.bond1.gc_stale_time = 60
net.ipv4.neigh.bond1.delay_first_probe_time = 5
net.ipv4.neigh.bond1.base_reachable_time = 30
net.ipv4.neigh.bond1.retrans_time = 99
net.ipv4.neigh.bond1.app_solicit = 0
net.ipv4.neigh.bond1.ucast_solicit = 3
net.ipv4.neigh.bond1.mcast_solicit = 3
net.ipv4.neigh.bond0.base_reachable_time_ms = 30000
net.ipv4.neigh.bond0.retrans_time_ms = 1000
net.ipv4.neigh.bond0.locktime = 99
net.ipv4.neigh.bond0.proxy_delay = 79
net.ipv4.neigh.bond0.anycast_delay = 99
net.ipv4.neigh.bond0.proxy_qlen = 64
net.ipv4.neigh.bond0.unres_qlen = 3
net.ipv4.neigh.bond0.gc_stale_time = 60
net.ipv4.neigh.bond0.delay_first_probe_time = 5
net.ipv4.neigh.bond0.base_reachable_time = 30
net.ipv4.neigh.bond0.retrans_time = 99
net.ipv4.neigh.bond0.app_solicit = 0
net.ipv4.neigh.bond0.ucast_solicit = 3
net.ipv4.neigh.bond0.mcast_solicit = 3
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.locktime = 99
net.ipv4.neigh.lo.proxy_delay = 79
net.ipv4.neigh.lo.anycast_delay = 99
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.unres_qlen = 3
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.base_reachable_time = 30
net.ipv4.neigh.lo.retrans_time = 99
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.locktime = 99
net.ipv4.neigh.default.proxy_delay = 79
net.ipv4.neigh.default.anycast_delay = 99
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.unres_qlen = 3
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.base_reachable_time = 30
net.ipv4.neigh.default.retrans_time = 99
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.udp_wmem_min = 4096
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_mem = 4651296 6201728 9302592
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_abc = 0
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_frto = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.icmp_ratemask = 6168
net.ipv4.icmp_ratelimit = 1000
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_rmem = 4194304 4194304 4194304
net.ipv4.tcp_wmem = 4194304 4194304 4194304
net.ipv4.tcp_mem = 196608 262144 393216
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_reordering = 127
net.ipv4.tcp_fack = 1
net.ipv4.tcp_orphan_retries = 0
net.ipv4.inet_peer_gc_maxtime = 120
net.ipv4.inet_peer_gc_mintime = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_max_memberships = 20
net.ipv4.route.rt_cache_rebuild_count = 4
net.ipv4.route.secret_interval = 600
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.error_burst = 5000
net.ipv4.route.error_cost = 1000
net.ipv4.route.redirect_silence = 20480
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_load = 20
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_timeout = 300
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.max_size = 8388608
net.ipv4.route.gc_thresh = 524288
net.ipv4.route.max_delay = 10
net.ipv4.route.min_delay = 2
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.ipfrag_time = 30
net.ipv4.ip_dynaddr = 0
net.ipv4.ipfrag_low_thresh = 196608
net.ipv4.ipfrag_high_thresh = 262144
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syn_retries = 5
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_forward = 0
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.netdev_budget = 300
net.core.somaxconn = 128
net.core.xfrm_larval_drop = 0
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_aevent_etime = 10
net.core.optmem_max = 20480
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_max_backlog = 1000
net.core.dev_weight = 64
net.core.rmem_default = 129024
net.core.wmem_default = 129024
net.core.rmem_max = 1048575
net.core.wmem_max = 1048575

NETSTAT:
Ip:
2561037608 total packets received
0 forwarded
0 incoming packets discarded
2560656723 incoming packets delivered
1259664034 requests sent out
30 dropped because of missing route
7540 reassemblies required
1508 packets reassembled ok
1508 fragments received ok
7540 fragments created
Icmp:
9023 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 34
echo requests: 8974
echo replies: 15
10860 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 1855
echo request: 31
echo replies: 8974
Tcp:
2283686 active connections openings
2288753 passive connection openings
7 failed connection attempts
29 connection resets received
586 connections established
2560547163 segments received
1259531668 segments send out
3186 segments retransmited
0 bad segments received.
67 resets sent

TcpExt:
2285835 TCP sockets finished time wait in fast timer
18378 time wait sockets recycled by time stamp
4359924 delayed acks sent
39965 delayed acks further delayed because of locked socket
Quick ack mode was activated 35469 times
41309168 packets directly queued to recvmsg prequeue.
204187372 packets directly received from backlog
582230058 packets directly received from prequeue
2338146696 packets header predicted
18911703 packets header predicted and directly queued to user
5841982 acknowledgments not containing data received
190644069 predicted acknowledgments
14 times recovered from packet loss due to SACK data
TCPDSACKUndo: 3
1167 congestion windows recovered after partial ack
0 TCP data loss events
138 timeouts after SACK recovery
15 fast retransmits
462 retransmits in slow start
1936 other TCP timeouts
3 times receiver scheduled too late for direct processing
35680 DSACKs sent for old packets
3763 DSACKs sent for out of order packets
2670 DSACKs received
6 connections reset due to unexpected data
6 connections reset due to early user close
5 connections aborted due to timeout
IpExt:
InBcastPkts: 14723
OutBcastPkts: 8271

CLIENT KERNEL:
net.unix.max_dgram_qlen = 10
net.token-ring.rif_timeout = 600000
net.ipv4.conf.eth5.promote_secondaries = 0
net.ipv4.conf.eth5.force_igmp_version = 0
net.ipv4.conf.eth5.disable_policy = 0
net.ipv4.conf.eth5.disable_xfrm = 0
net.ipv4.conf.eth5.accept_local = 0
net.ipv4.conf.eth5.arp_accept = 0
net.ipv4.conf.eth5.arp_ignore = 0
net.ipv4.conf.eth5.arp_announce = 0
net.ipv4.conf.eth5.arp_filter = 0
net.ipv4.conf.eth5.tag = 0
net.ipv4.conf.eth5.log_martians = 0
net.ipv4.conf.eth5.bootp_relay = 0
net.ipv4.conf.eth5.medium_id = 0
net.ipv4.conf.eth5.proxy_arp = 0
net.ipv4.conf.eth5.accept_source_route = 0
net.ipv4.conf.eth5.send_redirects = 1
net.ipv4.conf.eth5.rp_filter = 1
net.ipv4.conf.eth5.shared_media = 1
net.ipv4.conf.eth5.secure_redirects = 1
net.ipv4.conf.eth5.accept_redirects = 1
net.ipv4.conf.eth5.mc_forwarding = 0
net.ipv4.conf.eth5.forwarding = 0
net.ipv4.conf.bond1.promote_secondaries = 0
net.ipv4.conf.bond1.force_igmp_version = 0
net.ipv4.conf.bond1.disable_policy = 0
net.ipv4.conf.bond1.disable_xfrm = 0
net.ipv4.conf.bond1.accept_local = 0
net.ipv4.conf.bond1.arp_accept = 0
net.ipv4.conf.bond1.arp_ignore = 0
net.ipv4.conf.bond1.arp_announce = 0
net.ipv4.conf.bond1.arp_filter = 0
net.ipv4.conf.bond1.tag = 0
net.ipv4.conf.bond1.log_martians = 0
net.ipv4.conf.bond1.bootp_relay = 0
net.ipv4.conf.bond1.medium_id = 0
net.ipv4.conf.bond1.proxy_arp = 0
net.ipv4.conf.bond1.accept_source_route = 0
net.ipv4.conf.bond1.send_redirects = 1
net.ipv4.conf.bond1.rp_filter = 1
net.ipv4.conf.bond1.shared_media = 1
net.ipv4.conf.bond1.secure_redirects = 1
net.ipv4.conf.bond1.accept_redirects = 1
net.ipv4.conf.bond1.mc_forwarding = 0
net.ipv4.conf.bond1.forwarding = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.neigh.eth5.base_reachable_time_ms = 30000
net.ipv4.neigh.eth5.retrans_time_ms = 1000
net.ipv4.neigh.eth5.locktime = 99
net.ipv4.neigh.eth5.proxy_delay = 79
net.ipv4.neigh.eth5.anycast_delay = 99
net.ipv4.neigh.eth5.proxy_qlen = 64
net.ipv4.neigh.eth5.unres_qlen = 3
net.ipv4.neigh.eth5.gc_stale_time = 60
net.ipv4.neigh.eth5.delay_first_probe_time = 5
net.ipv4.neigh.eth5.base_reachable_time = 30
net.ipv4.neigh.eth5.retrans_time = 99
net.ipv4.neigh.eth5.app_solicit = 0
net.ipv4.neigh.eth5.ucast_solicit = 3
net.ipv4.neigh.eth5.mcast_solicit = 3
net.ipv4.neigh.bond1.base_reachable_time_ms = 30000
net.ipv4.neigh.bond1.retrans_time_ms = 1000
net.ipv4.neigh.bond1.locktime = 99
net.ipv4.neigh.bond1.proxy_delay = 79
net.ipv4.neigh.bond1.anycast_delay = 99
net.ipv4.neigh.bond1.proxy_qlen = 64
net.ipv4.neigh.bond1.unres_qlen = 3
net.ipv4.neigh.bond1.gc_stale_time = 60
net.ipv4.neigh.bond1.delay_first_probe_time = 5
net.ipv4.neigh.bond1.base_reachable_time = 30
net.ipv4.neigh.bond1.retrans_time = 99
net.ipv4.neigh.bond1.app_solicit = 0
net.ipv4.neigh.bond1.ucast_solicit = 3
net.ipv4.neigh.bond1.mcast_solicit = 3
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.locktime = 99
net.ipv4.neigh.lo.proxy_delay = 79
net.ipv4.neigh.lo.anycast_delay = 99
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.unres_qlen = 3
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.base_reachable_time = 30
net.ipv4.neigh.lo.retrans_time = 99
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.locktime = 99
net.ipv4.neigh.default.proxy_delay = 79
net.ipv4.neigh.default.anycast_delay = 99
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.unres_qlen = 3
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.base_reachable_time = 30
net.ipv4.neigh.default.retrans_time = 99
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.udp_wmem_min = 4096
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_mem = 1548960 2065280 3097920
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_abc = 0
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_frto = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.icmp_ratemask = 6168
net.ipv4.icmp_ratelimit = 1000
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_rmem = 4194304 4194304 4194304
net.ipv4.tcp_wmem = 4194304 4194304 4194304
net.ipv4.tcp_mem = 10485760 10485760 10485760
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_fack = 1
net.ipv4.tcp_orphan_retries = 0
net.ipv4.inet_peer_gc_maxtime = 120
net.ipv4.inet_peer_gc_mintime = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_max_memberships = 20
net.ipv4.route.rt_cache_rebuild_count = 4
net.ipv4.route.secret_interval = 600
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.error_burst = 5000
net.ipv4.route.error_cost = 1000
net.ipv4.route.redirect_silence = 20480
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_load = 20
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_timeout = 300
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.max_size = 8388608
net.ipv4.route.gc_thresh = 524288
net.ipv4.route.max_delay = 10
net.ipv4.route.min_delay = 2
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.ip_local_reserved_ports =
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.ipfrag_time = 30
net.ipv4.ip_dynaddr = 0
net.ipv4.ipfrag_low_thresh = 196608
net.ipv4.ipfrag_high_thresh = 262144
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syn_retries = 5
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_forward = 0
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.netdev_budget = 300
net.core.somaxconn = 128
net.core.xfrm_larval_drop = 0
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_aevent_etime = 10
net.core.optmem_max = 20480
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_max_backlog = 30000
net.core.dev_weight = 64
net.core.rmem_default = 129024
net.core.wmem_default = 129024
net.core.rmem_max = 131071
net.core.wmem_max = 131071

CLIENT MOUNT PARAMS:
type nfs (rw,remount,soft,intr,nfsvers=3,tcp,rsize=1048576,wsize=1048576,addr=10.52.115.30)

CLIENT NETSTAT -S (UP LESS THAN 1 HR):
Ip:
31389114 total packets received
1679 with invalid addresses
0 forwarded
0 incoming packets discarded
31387435 incoming packets delivered
18865576 requests sent out
33 dropped because of missing route
Icmp:
1177 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 459
echo requests: 715
echo replies: 3
1181 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 463
echo request: 3
echo replies: 715
IcmpMsg:
InType0: 3
InType3: 459
InType8: 715
OutType0: 715
OutType3: 463
OutType8: 3
Tcp:
1185 active connections openings
2087 passive connection openings
11 failed connection attempts
423 connection resets received
63 connections established
31288350 segments received
18758996 segments send out
2979 segments retransmited
0 bad segments received.
570 resets sent
Udp:
92551 packets received
463 packets to unknown port received.
0 packet receive errors
102422 packets sent
TcpExt:
153 invalid SYN cookies received
833 packets pruned from receive queue because of socket buffer overrun
1769 TCP sockets finished time wait in fast timer
3304 delayed acks sent
5 delayed acks further delayed because of locked socket
Quick ack mode was activated 63106 times
28130213 packets directly queued to recvmsg prequeue.
1707164504 packets directly received from backlog
41151849866 packets directly received from prequeue
630755 packets header predicted
29052459 packets header predicted and directly queued to user
17104 acknowledgments not containing data received
580376 predicted acknowledgments
100 times recovered from packet loss due to SACK data
Detected reordering 1 times using FACK
Detected reordering 6 times using SACK
87 congestion windows recovered after partial ack
1952 TCP data loss events
1830 fast retransmits
847 retransmits in slow start
145 other TCP timeouts
84214 packets collapsed in receive queue due to low socket buffer
63907 DSACKs sent for old packets
1143 DSACKs sent for out of order packets
392 DSACKs received
241 connections reset due to unexpected data
419 connections reset due to early user close
15 connections aborted due to timeout
IpExt:
InMcastPkts: 62
OutMcastPkts: 56
InBcastPkts: 4862