dear all :
I have some trouble with my linux server.
Dell R720
centos5.8 x64
8g merroy
4 cpu cores
Services:web (lnmp) and slave mysql
And now I find the disk io is so high and ssh to this server slowly.
this is some status as follows:
netstat:
Code:
[root@localhost ~]# netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'
TIME_WAIT 7198
FIN_WAIT1 324
FIN_WAIT2 92
ESTABLISHED 1008
SYN_RECV 187
LAST_ACK 3
After started block_dump:
Code:
[root@localhost ~]# dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
3880 kjournald
3087 nginx
1680 pdflush
735 mysqld
525 php
154 nfsd
94 check
31 irqbalance
30 crond
18 bash
memory:
Code:
[root@localhost ~]# free -m
total used free shared buffers cached
Mem: 7908 7852 55 0 318 1898
-/+ buffers/cache: 5636 2271
Swap: 8118 568 7550
this is the new add options from sysctl.conf
Code:
net.ipv4.tcp_max_syn_backlog = 65536
net.core.netdev_max_backlog = 32768
net.core.somaxconn = 32768
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_recycle = 1
#net.ipv4.tcp_tw_len = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 30
net.ipv4.ip_local_port_range = 1024 65535
kernel.shmmax = 128000000
iotop status,the nginx and kjournald 's disk io is so high some time:
iostat:
vmstat:
Code:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 581984 73452 323952 1855508 1 1 156 168 25 16 35 6 55 5 0
0 2 581984 73288 323968 1855592 0 0 288 56 1456 668 2 1 94 3 0
1 0 581984 73264 324008 1856352 0 0 664 0 5432 5711 35 6 55 4 0
15 0 581984 68828 324060 1856876 0 0 580 80 6862 7049 44 7 47 2 0
0 0 581984 71516 324068 1857156 0 0 200 0 8100 4989 65 7 28 0 0
0 0 581984 71456 324076 1857168 0 0 28 592 1387 255 6 0 91 3 0
2 0 581984 70800 324144 1857676 0 0 756 56 5184 4792 26 4 67 3 0
0 0 581984 67568 324184 1858340 0 0 436 0 9036 7082 70 9 20 0 0
2 0 581984 65088 324192 1858344 0 0 36 20 1812 797 8 1 90 0 0
1 0 581984 63988 324236 1858884 0 0 540 0 5650 5332 34 5 58 2 0
34 0 581984 56288 324264 1859132 0 0 432 692 7295 6227 49 8 41 2 0
0 0 581984 50376 324076 1855496 0 0 168 160 7811 5525 59 7 35 1 0
0 1 581984 68268 324100 1855620 0 0 104 0 1656 676 6 1 91 1 0
4 2 581984 64924 324160 1856276 0 0 956 8 5813 5566 52 7 37 5 0
2 0 581984 60252 324200 1857704 0 0 1208 0 8080 7327 60 8 28 3 0
0 0 581984 65284 324212 1857652 0 0 40 656 2388 1230 9 2 86 3 0
3 2 581984 64340 324280 1858436 0 0 1024 0 6365 5050 43 6 49 3 0
1 0 581984 62064 324344 1859604 0 0 1128 156 6151 5719 52 15 30 2 0
0 0 581984 59484 324400 1860196 0 0 612 0 1649 1231 13 5 77 4 0
5 1 581984 60916 324708 1860976 0 0 744 540 4652 4855 18 4 62 16 0
36 4 581984 49336 325040 1861368 0 0 600 1672 7069 6063 59 11 21 9 0
I think it maybe because of the disk io.but i can't find what the source of the problem.
Please give me some suggestion and solve it！
Thanks all!
