LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices

Reply
 
Search this Thread
Old 06-07-2013, 12:47 PM   #1
140fulton
LQ Newbie
 
Registered: Jun 2013
Posts: 5

Rep: Reputation: Disabled
IOSTAT double counting AIO write operations?


[I originally posted this in Linux-Server, but having reviewed the iostat sources it seems possible that it belongs here instead]


As part of debugging an IO generator, I've run across an odd set of results under RH 6.4 running on x64 hardware (validated on multiple systems):

-- The application uses async IO to access raw disk partitions (/dev/sdb1), and is run as root

-- The application reports 100 io/s, and 60% writes.

-- strace of the application reports 100 calls per second to io_submit() and io_getevents(), reflecting 100 AIO operations per second

-- 'dstat --aio' shows 100 ops per second

-- iostat reports 40 reads/sec, as expected, but 120 writes/second, twice as many as expected

Have you seen this behavior? Is it a known bug?

Thanks.

Jack
 
Old 06-07-2013, 03:03 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 1,531

Rep: Reputation: 374Reputation: 374Reputation: 374Reputation: 374
What size are your writes and what is the largest size allowed by the adapter and drive? SCSI will break transfers up when they are too large. Divide the MB/s reported by iostat by IOPS to see if this is happening.
 
Old 06-07-2013, 05:46 PM   #3
140fulton
LQ Newbie
 
Registered: Jun 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
smallpond,

A good suggestion -- and one I hadn't thought to check. The IOs vary in size for 4k - 64k, averaging about 8k, so I don't think they're being split (I've seen that for 1M IO, but not smaller). I did verify the bytes/IOP for one of streams in the workload, and it's spot-on and not being split. Thanks for the suggestion.

Jack
 
Old 06-10-2013, 03:51 PM   #4
140fulton
LQ Newbie
 
Registered: Jun 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
An update:

Comparing the output of the user-level application with a SAS protocol trace, it appears that IOSTAT is in fact reporting double the number of writes that are actually being issued. Now to find a way to "encourage" a bug fix...
 
Old 06-12-2013, 07:05 PM   #5
140fulton
LQ Newbie
 
Registered: Jun 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
Problem traced to CFQ scheduler

After some additional testing, it appears that this problem is linked to the (default) CFQ scheduler. Moving to the NOP scheduler brought IOSTAT in line with the other tools.
 
  


Reply

Tags
iostat, linux


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
IOSTAT double counting AIO write operations? 140fulton Linux - Server 1 06-08-2013 06:24 AM
How to write a wrapper script for iostat srrijith Linux - Newbie 20 12-10-2010 07:30 AM
Slow read/write operations on 3592 drive Joejr4u Linux - Hardware 2 03-17-2008 05:25 AM
kubuntu doing write operations Four Linux - Newbie 5 10-30-2006 01:11 PM


All times are GMT -5. The time now is 09:01 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration