LinuxQuestions.org - How to track which process is accessing disk?

- Linux - Kernel (https://www.linuxquestions.org/questions/linux-kernel-70/)

- - How to track which process is accessing disk? (https://www.linuxquestions.org/questions/linux-kernel-70/how-to-track-which-process-is-accessing-disk-505736/)

How to track which process is accessing disk?

I'm running Ubuntu Edgy on my laptop. It has 2GB RAM, so swapping should rarely happen. Indeed, my swap partition is almost always unused. However, the HD light is almost constantly on, only turn off from time to time for very short durations. This is not a problem of the HD light, because the HD activity monitor also shows frequent activity.

Could anyone tell me how could I continuously track which process is access the disk?

Thanks.

First thing I would do is check which processes are the most active (top). Decide if they are necessary for basic system operation or if they are fluff, unnecessary ballast, eye candy and such. Check if they are standalone processes or services, then shut them down appropriately to see if it helps. Chances are these processes are related to DE's or WM's you run or maybe [di]notify/fam/dbus-related. If that still isn't enough you could try finding out which files are opened for reading (r), writing (w) or both (u) using lsof. If handling lsof is a bit too daunting you could save the code below as "/tmp/showprocs.sh". Running it as "sh /tmp/showprocs.sh" shows filenames and "sh /tmp/showprocs.sh --procs" shows process names to investigate for fluff vs function.

Code:

#!/bin/sh

case "$1" in --pids) SEL=2;; --procs) SEL=1;;

--files|*) SEL=9;; # how come I can't use NF?

esac; for tree in home proc dev usr var; do 

echo -n "/$tree: "; lsof -w -n -a -d0-10 -a +D \

/$tree|grep '[0-9][urw]'|awk --assign VAR="$SEL" \

'{print $VAR}'|sort|uniq|xargs; done; exit 0

The only "easy" application I know that could show finegrained and continuous disk access details per application is "atop": needs a kernel patch though. Other ways to track disk access could be tracking syscalls with TrackFS or LTT but the output avalanche is overkill. I'm probably overlooking some alternatives and I don't know any GUI equivelents, hopefully somebody will chip in.

Also bear in mind that there are some activities that are scheduled via cron jobs, such as:

updatedb (which updates the slocate cache)
jobs (on some distros) which try to "prelink" executables
various and sundry cleanup and housekeeping tasks.

So, if you see a lot of disk activity, it's possible that it's simply a deferred job that hasn't had a chance to run. Or, it could be a cron job that is scheduled far too often. Does it keep doing it?

The system log, such as /var/log/messages, is a good place to find clues.

Thanks a lot for your help. I will try your advice later.

I've noticed in Debian with Gnome that my HD light will blink on and off continuously, but there is practically no disk activity, and it can happen at times when there is no swap in use at all. I suspect that it's from famd. I've never bothered to investigate because I think it's perfectly normal in this particular case and doesn't use up any noticeable resources. So, if it seems like your situation is similar, you may try killing famd and see if it quits doing it for you (assuming you're using fam in the first place).

Now when I use other distros, or WMs the behavior, that I outline, is not present. So I'm pretty sure that it's some daemon, service, or process that my version of gnome is running in the background to serve some purpose. I just never really cared enough to try to narrow it down.

I was never able to come up with a good way to track disk activity to which process (didn't have any luck in finding a program for this. Would be great if someone knows of a good program for this sort of thing.), but I did figure out which process is responsible for the HD LED blinking on consistently at 2 second intervals. It's 'hald' that's the cause of this.

What I can tell you about this is that it's not writing nor reading from any filesystem/hard drives when it's making the LED blink. I verified this through dstat. Also another thing that dstat revealed was that there is a write of exactly 24k every 5 second to the HD. I'm not 100%, but I'm pretty sure that this is nothing more than the journal of my ext3 filesystem(s) being updated.

The way in which I figured it out was just going through and killing off processes until the HD no longer did it's blinking (started with the more trivial and likely suspect of daemons and services). Far from scientific, and also a rather messy approach, but I finally got fed up and decided to take drastic measures like this. Killing off processes like this is probably not the best idea and likely to mess with the stability of the system (though it didn't in my case, but I've read that it's a good possibility). But if in your case, like mine, a reboot isn't a big deal, giving the system nice reboot afterwards, makes it a moot point whether this method is less than ideal. :)

Quote:

If handling lsof is a bit too daunting you could save the code below as "/tmp/showprocs.sh". Running it as "sh /tmp/showprocs.sh" shows filenames and "sh /tmp/showprocs.sh --procs" shows process names to investigate for fluff vs function.

The script seems to work OK, but it is really slow if there are many files on the disk.

Quote:

The only "easy" application I know that could show finegrained and continuous disk access details per application is "atop": needs a kernel patch though.

Tried atop, and is really nice, as it can list processes by disk usage, and also shows the amount of bytes read/written. Moreover, it didn't need any kernel patch, at least with Ubuntu 8.10's stock kernel (2.6.27-11-generic x86_64).