Visit Jeremy's Blog.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


Search this Thread
Old 07-27-2011, 01:05 PM   #1
Registered: Jul 2009
Posts: 47

Rep: Reputation: 0
Programming with kernel aio

I have written a program to test kernel aio functions (io_prep_pwrite, io_submit, io_getevents, etc. Note that this is not posix aio).

The test program creates a 2D array and then writes it to file using kernel aio functions. The way kernel aio works is that we first initialize some data structures, prepare them for writing (io_prep_pwrite), submit them (io_submit), wait for the i/o events to return (io_getevents) and finally call a callback function to ensure the writes are completed.

I am facing an issue at the callback function, which reports that something went wrong with the writes. It says that the events returned successfully, but 0 bytes were written. Also, the file written is of incorrect size. Below is the complete program.

 * File:   myaiocp.c
 * Author: kmehta
 * Created on July 11, 2011, 12:50 PM
 * Testing kernel aio. 
 * Program creates a 2D matrix and writes it multiple times to create a file of desired size. 
 * Writes are performed using kernel aio functions (io_prep_pwrite, io_submit, etc.)
#define _GNU_SOURCE
#define _XOPEN_SOURCE 600

#include <stdio.h>
#include <stdlib.h>
#include <getopt.h>
#include <pthread.h>
#include <fcntl.h>
#include <string.h>
#include <sys/uio.h>
#include <sys/time.h>
#include <omp.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <libaio.h>

char ** buf;
long seg_size;
int seg_rows;
double total_size;
char * filename;
static int wait_count = 0;

void io_task();
void cleanup();
void allocate_2D_matrix(int[]);
int file_open(char *);
void wr_done(io_context_t ctx, struct iocb* iocb, long res, long res2);

int main(int argc, char **argv) {
    total_size  = 1048576;      //1MB
    seg_size    = 1024;         //1kB
    seg_rows    = 1024;
    filename    = "/tmp/aio.out";

    int dims[] = {seg_rows, seg_size};
    allocate_2D_matrix(dims);   //Creates 2D matrix

    return 0;

 * Create a 2D matrix
void allocate_2D_matrix(int dims[2]) {
    int i;
    char *data;

    //create the matrix
    data = (char *) calloc(1, dims[0] * dims[1] * sizeof (char));
    if (data == NULL) {
        printf("\nCould not allocate memory for matrix.\n");

    buf = (char **) malloc(dims[0] * sizeof (char *));
    if (buf == NULL) {
        printf("\nCould not allocate memory for matrix.\n");

    for (i = 0; i < dims[0]; i++) {
        buf[i] = &(data[i * dims[1]]);

static void io_error(const char *func, int rc)
    if (rc == -ENOSYS)
        fprintf(stderr, "AIO not in this kernel\n");
    else if (rc < 0)
        fprintf(stderr, "%s: %s\n", func, strerror(-rc));
        fprintf(stderr, "%s: error %d\n", func, rc);


 * Callback function
static void work_done(io_context_t ctx, struct iocb *iocb, long res, long res2)

    if (res2 != 0) {
        io_error("aio write", res2);

      if (res != iocb->u.c.nbytes) {
            fprintf(stderr, "write missed bytes expect %lu got %ld\n",
                  iocb->u.c.nbytes, res2);
      wait_count --;
      printf("%d ", wait_count);

 * Wait routine. Get events and call the callback function work_done()
int io_wait_run(io_context_t ctx, long iter)
      struct io_event events[iter];
      struct io_event *ep;
      int ret, n;

       * get up to aio_maxio events at a time.
      ret = n = io_getevents(ctx, iter, iter, events, NULL);
      printf("got %d events\n", n);
       * Call the callback functions for each event.
      for (ep = events ; n-- > 0 ; ep++) {
            io_callback_t cb = (io_callback_t)ep->data ; struct iocb *iocb = ep->obj ; cb(ctx, iocb, ep->res, ep->res2);
      return ret;

void io_task() {
    long offset = 0;
    int bufIndex = 0;

    //Open file
    int fd = file_open(filename);

    //Initialize structures
    long i; 
    long iter = total_size / seg_size;  //No. of iterations to reach desired file size (total_size)
    io_context_t myctx;
    if(0 != io_queue_init(iter, &myctx))
        perror("Could not initialize io queue");
    struct iocb * ioq[iter];

    //loop through iter times to reach desired file size
    for (i = 0; i < iter; i++) {
        struct iocb *io = (struct iocb*) malloc(sizeof (struct iocb));
        io_prep_pwrite(io, fd, buf[bufIndex], seg_size, offset);
        io_set_callback(io, work_done);
        ioq[i] = io;

        offset += seg_size;
        bufIndex ++;
        if (bufIndex > seg_rows - 1)    //If entire matrix written, start again from index 0
            bufIndex = 0;

    printf("done preparing. Now submitting..\n");
    if(iter != io_submit(myctx, iter, ioq))
        perror("Failure on submit");

    printf("now awaiting completion..\n");
    wait_count = iter;
    int res;

    while (wait_count) {
        res = io_wait_run(myctx, iter);
        if (res < 0)
            io_error("io_wait_run", res);


void cleanup() {

int file_open(char *filename) {
    int fd;
    if (-1 == (fd = open(filename, O_DIRECT | O_CREAT | O_WRONLY | O_TRUNC, 0666))) {
        printf("\nError opening file. \n");

    return fd;
Thanks in advance.

Last edited by kvm1983; 07-27-2011 at 02:37 PM.
Old 07-27-2011, 01:53 PM   #2
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Q: Is work_done() getting invoked?

1. Put "printf's" in work_done(); print all relevant status variables each time it's called
2. Explicitly call aio_error(), "printf" the status of your I/O at key points
3. Post back the results

Great tutorial, if you're not already familiar with it:

Last edited by paulsm4; 07-27-2011 at 01:54 PM.
Old 07-27-2011, 02:37 PM   #3
Registered: Jul 2009
Posts: 47

Original Poster
Rep: Reputation: 0
Yes wr_done is getting called. In fact, the statement

if (res != iocb->u.c.nbytes) {
            fprintf(stderr, "write missed bytes expect %lu got %ld\n",
                  iocb->u.c.nbytes, res2);
triggers the error.

The link that you have posted talks about posix aio and not kernel aio. I should have differentiated in my original post, but I am not talking about posix aio functions.


inputoutput, kernel

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Difference between Userland programming and Kernel Programming.. meetsagittarian Programming 1 02-20-2006 12:26 AM
G3 AIO video out scheidel21 Linux - Hardware 0 10-28-2005 02:18 AM
Having problems with Kernel AIO with ASE Andrastus Linux - Enterprise 3 12-15-2004 07:53 AM
FastPath and AIO cdotger AIX 7 04-23-2004 03:56 AM
kernel and posix AIO suman_bahuguna Red Hat 0 04-01-2004 11:27 PM

All times are GMT -5. The time now is 02:53 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration