LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 10-02-2010, 01:24 PM   #1
mario.almeida
Member
 
Registered: May 2008
Location: India
Distribution: Ubuntu 10.04, CentOS
Posts: 179

Rep: Reputation: 27
Question High iowait%. How to find the cause?


OS: Red Hat Enterprise Linux ES release 4 (Nahant Update 4) (no support planned to upgrade to CentOS 5.5)
Linux AIR06 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux
Intel(R) Xeon(R) CPU 5160 @ 3.00GHz x 4
SWAP=/dev/sda6

Want to know the cause for high load average and iowait.
Want to confirm if I am running out of memory or the cause is something different?

Code:
top - 17:00:41 up 1057 days, 10:48,  4 users,  load average: 24.11, 16.06, 8.45
Tasks: 740 total,   1 running, 739 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.2% us,  5.6% sy,  6.3% ni, 33.0% id, 51.8% wa,  0.0% hi,  0.2% si
Mem:   8166092k total,  8125536k used,    40556k free,     3752k buffers
Swap:  8385888k total,  4264540k used,  4121348k free,    43000k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                         
  422 root      15   0     0    0    0 D    0  0.0 115:21.88 kjournald                                                       
 1998 root      15   0     0    0    0 D    0  0.0  67:25.16 kjournald                                                       
28473 root      16   0  3628  596  488 D    0  0.0   0:00.00 syslogd                                                         
 9740 root      15   0     0    0    0 D    0  0.0   0:00.00 pdflush                                                         
25344 root      15   0     0    0    0 D    0  0.0   0:00.00 pdflush                                                         
27329 root      18   0  5356  944  796 D    0  0.0   0:00.00 run-parts                                                       
Total status D: 6
Code:
free -o
             total       used       free     shared    buffers     cached
Mem:       8166092    8147336      18756          0       3752      43644
Swap:      8385888    4281388    4104500
Code:
iostat -kd 1
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             472.73      1074.75         0.00       1064          0
sda6            255.56      1022.22         0.00       1012          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              7.07        48.48         0.00         48          0
sda2              1.01         4.04         0.00          4          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             472.16       919.59        98.97        892         96
sda6            218.56       874.23         0.00        848          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3             24.74        37.11        74.23         36         72
sda2              8.25         8.25        24.74          8         24
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             455.10       457.14         0.00        448          0
sda6            114.29       457.14         0.00        448          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             407.14       424.49       363.27        416        356
sda6             91.84       367.35         0.00        360          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3             61.22        57.14       216.33         56        212
sda2             36.73         0.00       146.94          0        144
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             446.94      1228.57         0.00       1204          0
sda6            303.06      1212.24         0.00       1188          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              1.02        16.33         0.00         16          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             413.27      1077.55         8.16       1056          8
sda6            267.35      1069.39         0.00       1048          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              3.06         8.16         4.08          8          4
sda2              1.02         0.00         4.08          0          4
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             364.65      2533.33       404.04       2508        400
sda6            628.28      2513.13         0.00       2488          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3             59.60        20.20       222.22         20        220
sda2             45.45         0.00       181.82          0        180
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             510.20      6942.86         0.00       6804          0
sda6           1734.69      6938.78         0.00       6800          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              1.02         4.08         0.00          4          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             468.04      6103.09         0.00       5920          0
sda6           1525.77      6103.09         0.00       5920          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             476.77      6339.39         0.00       6276          0
sda6           1584.85      6339.39         0.00       6276          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             425.51      5648.98       179.59       5536        176
sda6           1412.24      5648.98         0.00       5536          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2             44.90         0.00       179.59          0        176
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             182.65      3085.71       412.24       3024        404
sda6            766.33      3065.31         0.00       3004          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3            108.16        20.41       412.24         20        404
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              18.81       277.23         0.00        280          0
sda6             69.31       277.23         0.00        280          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               7.14        69.39         0.00         68          0
sda6             17.35        69.39         0.00         68          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              16.33       265.31         0.00        260          0
sda6             66.33       265.31         0.00        260          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             135.35      2319.19       226.26       2296        224
sda6            579.80      2319.19         0.00       2296          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2             56.57         0.00       226.26          0        224
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              47.96       102.04       465.31        100        456
sda6             25.51       102.04         0.00        100          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3            116.33         0.00       465.31          0        456
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               4.08        57.14         0.00         56          0
sda6             14.29        57.14         0.00         56          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               9.18       142.86         0.00        140          0
sda6             35.71       142.86         0.00        140          0
sda5              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda1              0.00         0.00         0.00          0          0
 
Old 10-02-2010, 02:42 PM   #2
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,650

Rep: Reputation: 525Reputation: 525Reputation: 525Reputation: 525Reputation: 525Reputation: 525
Well, the processes in state 'D' drive up the load average and are in iowait. Active swapping could do that; so could a bad disk or a hung NFS mount. You should reduce your memory footprint or add RAM, so you swap less. You should also check for bad disks & wires.
 
1 members found this post helpful.
Old 10-02-2010, 02:50 PM   #3
mario.almeida
Member
 
Registered: May 2008
Location: India
Distribution: Ubuntu 10.04, CentOS
Posts: 179

Original Poster
Rep: Reputation: 27
Thumbs up

Thanks,

I was doubting the same thing.
 
  


Reply

Tags
iowait, performance


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
High IOWait issue misterphyrephox Linux - Newbie 6 01-06-2010 06:32 PM
high iowait in RHES sklam Red Hat 20 12-14-2006 01:35 PM
High Load, Low CPU/RAM/iowait ? newlinuxnewbie Linux - Server 1 09-22-2006 10:24 AM
High loadavg, low CPU & iowait reichhartkg Linux - General 1 07-28-2006 04:44 PM
CPUs in high IOwait state despite of lack of load kvsraju Linux - Enterprise 3 11-03-2005 04:34 PM


All times are GMT -5. The time now is 03:58 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration