LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-22-2015, 10:55 PM   #1
rainman1985_2010
Member
 
Registered: Oct 2010
Posts: 47

Rep: Reputation: 0
How to measure io latency?


Hi, everyone.

Recently, I have to measure the performance of my program, and have discovered that there is some performance bottle neck in it. My program is really io intensive, and I want to measure the latency of my IO operations to determine if this is where the bottle neck is.

My program uses libaio for I/O.

I measured the latency by calling the "times" function when before calling the "io_submit" and after calling the "io_getevents", and then compute the difference. But I wonder if this would lead to an unacceptable result because of the time consumed by "times" itself. And if so, how should I measure this latency?

Thanks:-)

Last edited by rainman1985_2010; 04-22-2015 at 11:13 PM.
 
Old 04-23-2015, 08:13 AM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,290

Rep: Reputation: 378Reputation: 378Reputation: 378Reputation: 378
How about running your program under a profiler like gprof? I haven't used it in awhile, but I suspect it would do the job. The profiler will give you a rough idea where your code is spending most of its line.

I you can categorize the I/O pattern of your program and want to get a system-level baseline for how well I/O is performing, you can use fio to benchmark your system's I/O performance. It has a mode that uses libaio, so the results ought to be pretty comparable. It reports bandwidth, latency, and CPU utilization during its run.
 
Old 04-24-2015, 01:16 PM   #3
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,914
Blog Entries: 13

Rep: Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948Reputation: 4948
Quote:
Originally Posted by rainman1985_2010 View Post
I measured the latency by calling the "times" function when before calling the "io_submit" and after calling the "io_getevents", and then compute the difference. But I wonder if this would lead to an unacceptable result because of the time consumed by "times" itself. And if so, how should I measure this latency?
I usually do something similar to exactly what you've done. I use the function which gets me seconds and milliseconds since the epoch and create logs. It's not uncommon to see a macro in my code that looks like a printf, but it's a "time logger" and the print strings are like "<1>\n" "<1a>\n" "<1b>\n" because I go from 1 to N and then find a low performing area, leave the remainder of my logs in there and then interpolate more granularity within the sub-area where I found too much time being taken. I make these macros conditional so that I can add them in or out depending on whether or not I need to re-check my performance.

I don't worry about the overhead in the prints or calls to time, it's worth more to determine where my problem areas are. The more main point here is I start granular and then fine tune. I.e. I get my whole initialization or data path via the 1, 2, 3, ... or A, B, C, ... method and when I find a particular area where it took a long time, I then drill deeper.

Also consider that if you add a timestamp just before loop logic (A) and then just after loop logic (A) and likely add one just before and after loop logic (B) then I've impacted the entry and exit for each of those sections similarly. And usually the exit timestamp for section (A) is the entry timestamp for section (B).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
21" TV: how do I measure it? stf92 General 21 08-03-2012 03:57 AM
How to measure disk latency? dbrazeau Linux - Software 1 02-17-2011 08:11 PM
how to measure interrupt latency using c in linux? vanu Linux - Newbie 2 04-23-2010 12:33 AM
Measure script mahmoud Programming 4 06-12-2009 03:41 PM
a way to measure how much flops your PC can do? pusrob Linux - Hardware 3 11-14-2007 11:54 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration