kernel 2.6.20 announced new real-time process IO statistics. Here is a fragment from the announcement:
I see no such thing in the several recent Linux distros I have tried out, with kernels up to 2.6.22. There are no /proc/<pid>/io directories (I did see one of these, but the disk numbers remained static).

There must be something special I must do to enable this feature. Where can I find information?

thanks

I am starting to look at this on 2.6.24-rc3
There is some reasonable doco in ../Documentation/accounting (from memory - I can't dial into my testbox from outside the office).

Looks like you're going to have to knock up some code to register an interest in the stats - then do your own maths. There is a sample shipped in the above location for delay stats.

I hope to get live stats the easy way, in /proc/pid/io, as advertised.

I saw some docs (taskstats.txt in kernel docs place, I think) about how to set up a socket to get live process stats with kernel queries and responses, but this was very skimpy and aimed at kernel pros. I was not able to understand enough of it - too many assumptions about prior knowledge.

I think the data is there, you just need to look in the right place. You have to make sure that if a command shells out, you look under that (new) PID, not the PID of the shell you used to launch it.
Also, once the task terminates the data is gone - at least as far as /proc/$PID/io is concerned anyway. If you have registered a listener, you can nominate to be notified when the watched task terminates - I haven't looked at that yet.

From the kernel config. you need to have CONFIG_TASKSTATS enabled.

Edit: I have just noticed we have had a similar thread previously.

Thanks.

None of the distros I have tried have any process IO data, for any PID.

I think CONFIG_TASKSTATS must be what I need. Is this a kernel compile time option?
Must I compile my own kernel to have this feature?

Yes - depending on the distro recompiling the kernel can be a serious pain. I just looked at this Ubuntu Grunty (7.10) config, and it is disabled.
Without this option enabled, you won't even see the /proc/$PID/io, so there is nothing you can do in that case.

As of today's release, if you've enabled process I/O stats in your kernel, collectl will display them. Check out http://collectl.sourceforge.net/ProcessIOStats.html and if you like what you see, download a copy.
btw - collectl does a LOT more than just show process stats and you can read more about it on the home page.
-mark

Looks mighty handy Mark - I'll have to have a play ...
Expect to hear from me ...

As an update I've just released a new version - see http://collectl.sourceforge.net/, which allows you to monitor interrupts by cpu, something I don't believe is available with any other tools! here's an example of one of the output formats, there are others that show even more details at http://collectl.sourceforge.net/Interrupts.html

#Time Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7
12:49:55 4828 16K 1000 16K 18 16K 16K 0
12:49:56 4804 16K 1000 16K 0 16K 16K 0
12:49:57 4811 16K 1000 16K 18 16K 16K 0
12:49:58 4789 16K 1000 16K 0 17K 16K 44

-mark

G'day Mark, I did look at your efforts. Nice.
For my particular needs, blktrace probably fits the bill better.

More tools is always better though - each situation (usually) needs something different.

I just wanted to let people know that I just released a new version of collectl at http://collectl.sourceforge.net/. Entering the command "collectl --top io" which show a dynamic list of processes sorted by I/O usage. If you add the switch "--procopts t" it will include threads. You can/should read more about monitoring processes at http://collectl.sourceforge.net/Process.html and also remember that collectl can do a lot more than just look at processes.

enjoy
-mark

Thanks Mark, I did pass by a few days back.
Will download it and have a look.

Cheers ...

These process IO counters are not available in any official RedHat or SuSE kernels, AFAIK. Does anyone know what the plans are to make them available? Or if they are already available, which versions?