IOSTAT double counting AIO write operations?
 

[I originally posted this in Linux-Server, but having reviewed the iostat sources it seems possible that it belongs here instead]


As part of debugging an IO generator, I've run across an odd set of results under RH 6.4 running on x64 hardware (validated on multiple systems):

-- The application uses async IO to access raw disk partitions (/dev/sdb1), and is run as root

-- The application reports 100 io/s, and 60% writes.

-- strace of the application reports 100 calls per second to io_submit() and io_getevents(), reflecting 100 AIO operations per second

-- 'dstat --aio' shows 100 ops per second

-- iostat reports 40 reads/sec, as expected, but 120 writes/second, twice as many as expected

Have you seen this behavior? Is it a known bug?

Thanks.

Jack

What size are your writes and what is the largest size allowed by the adapter and drive? SCSI will break transfers up when they are too large. Divide the MB/s reported by iostat by IOPS to see if this is happening.

smallpond,

A good suggestion -- and one I hadn't thought to check. The IOs vary in size for 4k - 64k, averaging about 8k, so I don't think they're being split (I've seen that for 1M IO, but not smaller). I did verify the bytes/IOP for one of streams in the workload, and it's spot-on and not being split. Thanks for the suggestion.

Jack

An update:

Comparing the output of the user-level application with a SAS protocol trace, it appears that IOSTAT is in fact reporting double the number of writes that are actually being issued. Now to find a way to "encourage" a bug fix...

Problem traced to CFQ scheduler
 

After some additional testing, it appears that this problem is linked to the (default) CFQ scheduler. Moving to the NOP scheduler brought IOSTAT in line with the other tools.