LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
Reload this Page IO Issues - Questions about iotop and top
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-30-2012, 10:38 AM   #1
trackpads
LQ Newbie
 
Registered: Apr 2011
Posts: 10

Rep: Reputation: 0
IO Issues - Questions about iotop and top

Hey,

I have a dedicated server but am having some serious backend issues. It is a discussion forum and gallery mostly that run on the server with a 16gb mysql database. The load never goes above 5 but the server is dog slow. I request a ram upgrade to 16gb and a SSD drive added as a secondary drive for the mysql tables. Attached is my top and iotop out puts, it wasnt that busy when I took them though. Could I be missing something or is it all mysql doing this?

Thanks again for the help,

-Jason
Attached Thumbnails
Click image for larger version Name: SP32-20121030-102830.jpg Views: 27 Size: 111.6 KB ID: 11099   Click image for larger version Name: SP32-20121030-102805.jpg Views: 22 Size: 139.9 KB ID: 11100  
 
Old 10-30-2012, 11:22 AM   #2
evgenyz
Member
 
Registered: Sep 2012
Posts: 48

Rep: Reputation: Disabled
Try to run iostat -k 2 20 and see the i/o distribution accross disks.
Does the networking involved in the processing as well? It can slow down the performance as well...
Also what is the speed of your drives?
 
Old 10-31-2012, 02:21 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,126

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
I can't spell mysql, but those numbers look seriously out of whack.
How can a system only using 168k of swap have tasks waiting on swapin ?. iotop uses the numbers from taskstats, so I'd presume they are correct, and kswapd is certainly being kicked into action. All bad signs I'd say.
Certainly looks like disk(s) are getting thrashed from those iowait numbers. Database and swap on the same disk(s) by any chance ?.

Which leaves mysql - maybe fixing buffers in RAM so it looks like normal "cached", but isn't as noone else can use it. I'm sure I've read something on Oracle (SGA ?) causing similar symptoms.
 
Old 11-01-2012, 10:09 AM   #4
trackpads
LQ Newbie
 
Registered: Apr 2011
Posts: 10

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by evgenyz View Post
Try to run iostat -k 2 20 and see the i/o distribution accross disks.
Does the networking involved in the processing as well? It can slow down the performance as well...
Also what is the speed of your drives?
Here is what I got, I am doing a SQL operation that is quite heavy to show the load:

iostat -k 2 20

Code:
Linux 2.6.18-308.16.1.el5 (caesar.trackpads.net)        11/01/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.75    0.01    6.45    2.88    0.00   64.92

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             114.70      1952.64      1216.59  321854573  200531322
sda1              0.00         0.01         0.00        996          2
sda2            114.70      1952.64      1216.59  321853436  200531320
dm-0            375.75      1950.43      1164.25  321489397  191903556
dm-1             13.64         2.20        52.34     363409    8627596
dm-2              0.00         0.00         0.00        496        168

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.50    0.50    2.19   38.81    0.00   46.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             444.50       714.00     11432.00       1428      22864
sda1              0.00         0.00         0.00          0          0
sda2            444.50       714.00     11432.00       1428      22864
dm-0           2937.50       800.00     11432.00       1600      22864
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.68    1.12    2.50   37.66    0.00   48.03

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             546.50       864.00     11856.00       1728      23712
sda1              0.00         0.00         0.00          0          0
sda2            546.50       864.00     11856.00       1728      23712
dm-0           3118.00       780.00     12200.00       1560      24400
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.86    0.75    2.56   40.32    0.00   45.51

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             481.50       742.00     11100.00       1484      22200
sda1              0.00         0.00         0.00          0          0
sda2            481.50       742.00     11100.00       1484      22200
dm-0           2790.00       816.00     10836.00       1632      21672
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.06    0.19    3.00   42.44    0.00   41.31

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             440.50       540.00     10440.00       1080      20880
sda1              0.00         0.00         0.00          0          0
sda2            440.50       540.00     10440.00       1080      20880
dm-0           2649.00       508.00     10360.00       1016      20720
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.30    0.06    3.69   51.09    0.00   29.86

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             474.50       420.00     12280.00        840      24560
sda1              0.00         0.00         0.00          0          0
sda2            474.50       420.00     12280.00        840      24560
dm-0           3133.50       414.00     12332.00        828      24664
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.25    0.00    2.31   48.44    0.00   38.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             486.00       446.00     13024.00        892      26048
sda1              0.00         0.00         0.00          0          0
sda2            486.00       446.00     13024.00        892      26048
dm-0           3290.00       474.00     12972.00        948      25944
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.25    0.00    3.31   34.31    0.00   47.12

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             571.50       570.00     15072.00       1140      30144
sda1              0.00         0.00         0.00          0          0
sda2            571.50       570.00     15072.00       1140      30144
dm-0           3877.50       552.00     15298.00       1104      30596
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.80    0.00    3.56   35.60    0.00   42.04

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             568.66       400.00     17568.16        804      35312
sda1              0.00         0.00         0.00          0          0
sda2            568.66       400.00     17568.16        804      35312
dm-0           4388.56       362.19     17399.00        728      34972
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.05    0.00    8.62   38.48    0.00   34.85

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             491.50       512.00     13106.00       1024      26212
sda1              0.00         0.00         0.00          0          0
sda2            491.50       512.00     13106.00       1024      26212
dm-0           3368.50       504.00     13268.00       1008      26536
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.26    0.00    4.88   36.02    0.00   40.84

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             572.00       676.00     14310.00       1352      28620
sda1              0.00         0.00         0.00          0          0
sda2            572.00       676.00     14310.00       1352      28620
dm-0           3598.50       720.00     14128.00       1440      28256
dm-1              0.50         2.00         0.00          4          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.49    0.00    2.43   37.45    0.00   49.63

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             504.50       816.00     12896.00       1632      25792
sda1              0.00         0.00         0.00          0          0
sda2            504.50       816.00     12896.00       1632      25792
dm-0           3271.50       784.00     12820.00       1568      25640
dm-1             10.00         0.00        40.00          0         80
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.37    0.00    2.94   29.75    0.00   52.94

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             538.50       724.00     13974.00       1448      27948
sda1              0.00         0.00         0.00          0          0
sda2            538.50       724.00     13974.00       1448      27948
dm-0           3577.50       734.00     14046.00       1468      28092
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.69    0.00    2.12   37.56    0.00   50.62

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             471.50       692.00     11470.00       1384      22940
sda1              0.00         0.00         0.00          0          0
sda2            471.50       692.00     11470.00       1384      22940
dm-0           2942.50       662.00     11526.00       1324      23052
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.61    0.00    3.37   29.48    0.00   50.53

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             599.00       494.00     15236.00        988      30472
sda1              0.00         0.00         0.00          0          0
sda2            599.00       494.00     15236.00        988      30472
dm-0           3835.50       498.00     15140.00        996      30280
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.79    0.00    6.74   37.58    0.00   36.89

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             587.50       556.00     15218.00       1112      30436
sda1              0.00         0.00         0.00          0          0
sda2            587.50       556.00     15218.00       1112      30436
dm-0           3899.00       560.00     15336.00       1120      30672
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.12    0.00    4.38   41.62    0.00   39.88

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             487.50       368.00     15654.00        736      31308
sda1              0.00         0.00         0.00          0          0
sda2            487.50       368.00     15654.00        736      31308
dm-0           4029.00       406.00     15950.00        812      31900
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.80    0.00    6.43   45.35    0.00   31.42

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             582.09       292.54     16931.34        588      34032
sda1              0.00         0.00         0.00          0          0
sda2            582.09       292.54     16931.34        588      34032
dm-0           4229.85       312.44     16805.97        628      33780
dm-1             15.42         1.99        59.70          4        120
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.55    0.00    3.00   39.41    0.00   43.04

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             557.50       284.00     16846.00        568      33692
sda1              0.00         0.00         0.00          0          0
sda2            557.50       284.00     16846.00        568      33692
dm-0           4199.00       218.00     16718.00        436      33436
dm-1              0.50         0.00         2.00          0          4
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.24    0.00    3.81   36.35    0.00   41.60

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             594.50       516.00     16026.00       1032      32052
sda1              0.00         0.00         0.00          0          0
sda2            594.50       516.00     16026.00       1032      32052
dm-0           3987.00       550.00     15772.00       1100      31544
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0
 
Old 11-03-2012, 03:55 PM   #5
markseger
Member
 
Registered: Jul 2003
Posts: 244

Rep: Reputation: 26
you're never going to tell what's wrong by looking at i/o rates as the real secret is in looking at what's happening at lower levels such as the disk service and wait times. One thing that's really easy to get fooled by is a disk that's not very busy. If it's doing random I/O it doesn't take a whole lot to saturate it. iostat -x will show that level of detail but quite honestly I can't handle reading the display - too much going on.

try collectl -sD --home and you'll see something much easier to read. if you have continual wait times over more than 100 msec or so, it could be that your disk(s) are flat out and nothing is going to help short of getting more disk or finding some other way to reduce the load on them.

-mark
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: iotop: Per Process I/O Usage LXer Syndicated Linux News 0 05-19-2011 09:30 AM
[SOLVED] anybody knows how to install iotop alexx2010 Linux - Newbie 2 10-20-2010 07:10 PM
Is there an iotop based on file access? MikeyCarter Linux - Software 1 08-18-2010 04:47 PM
Disk I/O stats (iotop alternative?) omnivir Linux - Enterprise 1 01-25-2010 10:07 AM
Is there a iotop which reports top file io? MikeyCarter Linux - Software 1 06-25-2009 12:27 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 01:03 PM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration