Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
| Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
07-23-2012, 12:25 PM
|
#1
|
|
Member
Registered: Mar 2010
Posts: 122
Rep: 
|
disk is thrashing (iostat, lsof, ps)
Greetings --I've noticed one of the drives on a long-time running box is pretty active. Given my initial notice was based on visual blinky lights and just hearing the disk grind, I have some output below that gives a general indication of activity.
Can you help me zero in on the issue? That is, what's causing this IO on /dev/sdc1 (mounted on /opt)?
Code:
mogwai:/root>iostat -x
Linux 2.6.18-308.1.1.el5 (mogwai) 07/23/2012
avg-cpu: %user %nice %system %iowait %steal %idle
0.80 0.98 1.52 0.45 0.00 96.26
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.28 15.36 0.93 4.98 56.54 162.75 37.13 0.04 6.03 0.31 0.18
sda1 0.02 0.00 0.00 0.00 0.04 0.00 22.17 0.00 1.14 0.95 0.00
sda2 0.01 0.00 0.00 0.00 0.03 0.00 52.36 0.00 0.36 0.29 0.00
sda3 0.24 15.36 0.92 4.98 56.46 162.75 37.13 0.04 6.03 0.31 0.18
sdb 0.01 0.00 0.00 0.00 0.05 0.00 34.71 0.00 4.59 4.12 0.00
sdb1 0.00 0.00 0.00 0.00 0.02 0.00 35.13 0.00 3.35 3.35 0.00
sdb2 0.00 0.00 0.00 0.00 0.02 0.00 33.67 0.00 2.75 2.75 0.00
sdc 0.27 2.15 3.54 5.44 89.35 109.88 22.17 0.11 12.53 1.95 1.75
sdc1 0.27 2.15 3.54 5.44 89.34 109.88 22.16 0.11 12.53 1.95 1.75
Code:
mogwai:/root>lsof | grep sdc
mogwai:/root>lsof | grep opt
smbd 3264 root mem REG 8,3 34240 10917347 /usr/lib64/libpopt.so.0.0.0
nmbd 3267 root mem REG 8,3 34240 10917347 /usr/lib64/libpopt.so.0.0.0
smbd 3278 root mem REG 8,3 34240 10917347 /usr/lib64/libpopt.so.0.0.0
smbd 12126 nobody mem REG 8,3 34240 10917347 /usr/lib64/libpopt.so.0.0.0
smbd 23573 root cwd DIR 8,33 4096 56090625 /opt/Software
smbd 23573 root mem REG 8,3 34240 10917347 /usr/lib64/libpopt.so.0.0.0
Code:
mogwai:/root>top -n 1
top - 10:23:50 up 12:24, 5 users, load average: 0.00, 0.01, 0.06
Tasks: 168 total, 1 running, 167 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 1.5%sy, 1.0%ni, 96.3%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8079076k total, 2919700k used, 5159376k free, 503208k buffers
Swap: 4096564k total, 0k used, 4096564k free, 2021932k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 10368 684 572 S 0.0 0.0 0:00.56 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.19 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.03 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
11 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/3
12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
14 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0
15 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1
16 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/2
17 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/3
18 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
87 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
94 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
95 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 kblockd/1
96 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/2
97 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/3
98 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
213 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
214 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/1
215 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/2
216 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/3
219 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
221 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
313 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
314 root 23 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
315 root 15 0 0 0 0 S 0.0 0.0 0:00.68 pdflush
316 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd0
317 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
318 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 aio/1
319 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 aio/2
320 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 aio/3
|
|
|
|
|
07-23-2012, 12:29 PM
|
#2
|
|
Member
Registered: Jun 2007
Distribution: Debian, SLES, Ubuntu
Posts: 308
Rep:
|
The information that you have provided so far doesn't show that the system is all that busy. Is this abnormal behavior for this system?
You might try running lsof /opt and giving us that output. Also what services are running on this box?
|
|
|
|
|
07-23-2012, 12:36 PM
|
#3
|
|
Member
Registered: Mar 2010
Posts: 122
Original Poster
Rep:
|
/opt is pretty much a read disk. That earlier lsof output was showing quite a few reads and writes for this disk. I use it as an mp3 jukebox and VirtualBox vms. Just moments ago I stopped quite a few procs and tried to manually unmount the disk. Without fuser or lsof reporting what was busy it refused to mount. I then uncommented the mount from /etc/fstab and restarted the system. Here is what things look like now:
Code:
avg-cpu: %user %nice %system %iowait %steal %idle
1.42 0.83 0.57 0.47 0.00 96.70
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 51.30 1912.29 182.18 564928 53820
sda1 0.19 5.73 0.01 1692 4
sda2 0.10 4.96 0.00 1466 0
sda3 50.97 1900.41 182.17 561418 53816
sdb 0.23 7.99 0.00 2360 0
sdb1 0.08 2.74 0.00 808 0
sdb2 0.08 2.74 0.00 808 0
sdc 0.36 6.75 0.03 1994 8
sdc1 0.30 4.39 0.03 1298 8
I will now start procs up one by one to see what activity resurfaces.
I guess my question is most specifically --Given the details above, how could one clue in on a particular process doing the reads/writes. lsof didn't seem to do that. |
|
|
|
|
07-23-2012, 12:57 PM
|
#4
|
|
Member
Registered: Jun 2007
Distribution: Debian, SLES, Ubuntu
Posts: 308
Rep:
|
lsof should be able to show you what files are in use in the location. Once you have that one of the fields is a PID. Using the PID you can start to trace back to what is going on with that PID and get an idea of what is going on. Just something that you have to research piece by piece to get complete information.
|
|
|
|
|
07-23-2012, 06:34 PM
|
#5
|
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,385
|
iotop
|
|
|
|
All times are GMT -5. The time now is 06:19 PM.
|
|
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
|
 Latest Threads
LQ News
|
|
