LinuxQuestions.org - What is the best way to investigate I/O performance issues?

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - What is the best way to investigate I/O performance issues? (https://www.linuxquestions.org/questions/linux-newbie-8/what-is-the-best-way-to-investigate-i-o-performance-issues-948711/)

What is the best way to investigate I/O performance issues?

All,

I have been reading about I/O monitoring using the commands like iostat, I see the numbers as below.But how to exactly determine at what numbers of tps and blks_read/s, blks_write/s, blk_read and blk_wrtn that I should be worrying about my I/O?

avg-cpu: %user %nice %system %iowait %steal %idle
18.47 0.00 3.37 1.50 0.00 75.79

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 5.13 54.42 62.16 716429654 818324689
sdb 1.90 53.31 33.36 701766492 439161792
sdc 20.13 115.79 0.61 1524333548 8085448
sdd 17.34 159.19 305.05 2095616964 4015769568
sde 94.33 305.71 194.14 4024341714 2555628160
sdf 0.00 0.00 0.00 45536 0
sdg 1.39 121.54 325.97 1600003486 4291054424
dm-0 0.18 20.10 1.01 264637438 13332840
dm-2 1.19 1.01 9.38 13306122 123450464
dm-3 80.15 0.37 314.80 4925530 4144126664
dm-4 0.51 0.09 4.01 1170674 52849384
dm-5 0.05 0.05 0.35 596386 4580104
dm-6 0.08 0.06 0.63 782970 8332944
dm-7 7.46 16.56 34.04 218062994 448063712
dm-8 8.03 61.96 53.75 815686650 707520240
dm-9 3.83 147.63 23.32 1943456178 306931096

Best Regards,
gdtm86

Quote:

Originally Posted by gdtm86@gmail.com (Post 4696405)

But how to exactly determine at what numbers of tps and blks_read/s, blks_write/s, blk_read and blk_wrtn that I should be worrying about my I/O?

avg-cpu: %user %nice %system %iowait %steal %idle
18.47 0.00 3.37 1.50 0.00 75.79

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 5.13 54.42 62.16 716429654 818324689
sdb 1.90 53.31 33.36 701766492 439161792

I believe the key here is iowait. It shows the percentage of time that the cpu is sitting idle, while it is waiting for an outstanding i/o request. To quote from this blog post...

Quote:

...is to determine whether an application is cpu-bound / memory-bound / IO bound. IO bound applications end up wasting cpu cycles, especially incase of Disk IO. Therefore in order to maximize performance and optimize resource utilization one should try and reduce iowait time of a CPU and tweak a deployment to make an application cpu-bound.

So I suggest that you plot %iowait against time. And if your iowait stays low, then your disk i/o perf. is okay. But if iowait increases, then your cpu is waiting for disk i/o to catch up. And that is when you need to start to worry about your disk i/o performance.

You can use atop. It'll show you where things are bottlenecking.

Thank you dru8274.

I read the link you posted and it is a nice article. It looks like I should be watching for %iowait and then also definitely look at the %util of the specific block device to see the bandwidth utilization.

Also Is there a way to compare the number of I/O operations(transactions)/sec on the device to max operations(transactions)/sec a device(scsi,ATA,ssd)can handle? I was wondering that could be a good indicator to see how busy the device is.

Look at the doco for iostat - it'll give you everything you need.

Note that these are all averages, and iowait doesn't necessarily imply there is actually any process truly waiting on I/O. Merely that the CPU(s) was/were idle and there was I/O still to complete.
There may be a correlation, but it isn't guaranteed.

You might want to activate sar http://linux.die.net/man/1/sar to collect/report statistics.