LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-23-2012, 01:25 PM   #1
ezekieldas
Member
 
Registered: Mar 2010
Posts: 122

Rep: Reputation: 16
disk is thrashing (iostat, lsof, ps)


Greetings --I've noticed one of the drives on a long-time running box is pretty active. Given my initial notice was based on visual blinky lights and just hearing the disk grind, I have some output below that gives a general indication of activity.

Can you help me zero in on the issue? That is, what's causing this IO on /dev/sdc1 (mounted on /opt)?

Code:
mogwai:/root>iostat -x
Linux 2.6.18-308.1.1.el5 (mogwai) 	07/23/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.80    0.98    1.52    0.45    0.00   96.26

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.28    15.36  0.93  4.98    56.54   162.75    37.13     0.04    6.03   0.31   0.18
sda1              0.02     0.00  0.00  0.00     0.04     0.00    22.17     0.00    1.14   0.95   0.00
sda2              0.01     0.00  0.00  0.00     0.03     0.00    52.36     0.00    0.36   0.29   0.00
sda3              0.24    15.36  0.92  4.98    56.46   162.75    37.13     0.04    6.03   0.31   0.18
sdb               0.01     0.00  0.00  0.00     0.05     0.00    34.71     0.00    4.59   4.12   0.00
sdb1              0.00     0.00  0.00  0.00     0.02     0.00    35.13     0.00    3.35   3.35   0.00
sdb2              0.00     0.00  0.00  0.00     0.02     0.00    33.67     0.00    2.75   2.75   0.00
sdc               0.27     2.15  3.54  5.44    89.35   109.88    22.17     0.11   12.53   1.95   1.75
sdc1              0.27     2.15  3.54  5.44    89.34   109.88    22.16     0.11   12.53   1.95   1.75
Code:
mogwai:/root>lsof | grep sdc
mogwai:/root>lsof | grep opt
smbd       3264      root  mem       REG                8,3    34240   10917347 /usr/lib64/libpopt.so.0.0.0
nmbd       3267      root  mem       REG                8,3    34240   10917347 /usr/lib64/libpopt.so.0.0.0
smbd       3278      root  mem       REG                8,3    34240   10917347 /usr/lib64/libpopt.so.0.0.0
smbd      12126    nobody  mem       REG                8,3    34240   10917347 /usr/lib64/libpopt.so.0.0.0
smbd      23573      root  cwd       DIR               8,33     4096   56090625 /opt/Software
smbd      23573      root  mem       REG                8,3    34240   10917347 /usr/lib64/libpopt.so.0.0.0
Code:
mogwai:/root>top -n 1
top - 10:23:50 up 12:24,  5 users,  load average: 0.00, 0.01, 0.06
Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  1.5%sy,  1.0%ni, 96.3%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8079076k total,  2919700k used,  5159376k free,   503208k buffers
Swap:  4096564k total,        0k used,  4096564k free,  2021932k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                
    1 root      15   0 10368  684  572 S  0.0  0.0   0:00.56 init                                   
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.19 migration/0                            
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0                            
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                             
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.03 migration/1                            
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1                            
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                             
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 migration/2                            
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2                            
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                             
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.01 migration/3                            
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3                            
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                             
   14 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0                               
   15 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/1                               
   16 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/2                               
   17 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/3                               
   18 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper                                
   87 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread                                
   94 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0                              
   95 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 kblockd/1                              
   96 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/2                              
   97 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/3                              
   98 root      17  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid                                 
  213 root      17  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0                               
  214 root      18  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/1                               
  215 root      18  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/2                               
  216 root      19  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/3                               
  219 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd                                  
  221 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod                                
  313 root      15   0     0    0    0 S  0.0  0.0   0:00.00 khungtaskd                             
  314 root      23   0     0    0    0 S  0.0  0.0   0:00.00 pdflush                                
  315 root      15   0     0    0    0 S  0.0  0.0   0:00.68 pdflush                                
  316 root      19  -5     0    0    0 S  0.0  0.0   0:00.00 kswapd0                                
  317 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0                                  
  318 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 aio/1                                  
  319 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 aio/2                                  
  320 root      12  -5     0    0    0 S  0.0  0.0   0:00.00 aio/3
 
Old 07-23-2012, 01:29 PM   #2
grim76
Member
 
Registered: Jun 2007
Distribution: Debian, SLES, Ubuntu
Posts: 308

Rep: Reputation: 50
The information that you have provided so far doesn't show that the system is all that busy. Is this abnormal behavior for this system?

You might try running lsof /opt and giving us that output. Also what services are running on this box?
 
Old 07-23-2012, 01:36 PM   #3
ezekieldas
Member
 
Registered: Mar 2010
Posts: 122

Original Poster
Rep: Reputation: 16
/opt is pretty much a read disk. That earlier lsof output was showing quite a few reads and writes for this disk. I use it as an mp3 jukebox and VirtualBox vms. Just moments ago I stopped quite a few procs and tried to manually unmount the disk. Without fuser or lsof reporting what was busy it refused to mount. I then uncommented the mount from /etc/fstab and restarted the system. Here is what things look like now:

Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.42    0.83    0.57    0.47    0.00   96.70

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              51.30      1912.29       182.18     564928      53820
sda1              0.19         5.73         0.01       1692          4
sda2              0.10         4.96         0.00       1466          0
sda3             50.97      1900.41       182.17     561418      53816
sdb               0.23         7.99         0.00       2360          0
sdb1              0.08         2.74         0.00        808          0
sdb2              0.08         2.74         0.00        808          0
sdc               0.36         6.75         0.03       1994          8
sdc1              0.30         4.39         0.03       1298          8
I will now start procs up one by one to see what activity resurfaces.

I guess my question is most specifically --Given the details above, how could one clue in on a particular process doing the reads/writes. lsof didn't seem to do that.
 
Old 07-23-2012, 01:57 PM   #4
grim76
Member
 
Registered: Jun 2007
Distribution: Debian, SLES, Ubuntu
Posts: 308

Rep: Reputation: 50
lsof should be able to show you what files are in use in the location. Once you have that one of the fields is a PID. Using the PID you can start to trace back to what is going on with that PID and get an idea of what is going on. Just something that you have to research piece by piece to get complete information.
 
Old 07-23-2012, 07:34 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 14,838

Rep: Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822
iotop
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
iostat to print disk usage speed arp99 Linux - Newbie 8 08-01-2011 11:12 PM
iostat is not working ?? only shows cpu, no disk stats ? BigCohunes Linux - Newbie 2 02-17-2009 08:35 AM
Software to find process responsible for disk thrashing JoeDuncan Linux - Software 3 08-02-2007 04:15 PM
iostat help.no disk stats droberts Linux - Newbie 0 11-09-2005 04:36 PM
disk thrashing at 4am Pete Dogg Red Hat 7 11-13-2003 05:43 PM


All times are GMT -5. The time now is 11:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration