file writes to RAID using posix_fadvise vs O_DIRECT
 

I would extremely benefit from knowing which way is best. I have tried to utilize posix_fadvise(fd,pos,size,POSIX_FADV_DONTNEED) and in another instance O_DIRECT, but both still use the cache unneccessarily.

I have a specific case where I have a device driver that has multiple pci_alloc_consistent buffers that it has mapped into user space through MMAP. Then, my user app maps into those buffers (each 4MB) and then needs to recursively call a file write to send the data over to a striped RAID fully capable of throughputs above 200MB/s, but I only receive 70-90 MB/s. I don't need to look at the data I just accessed in the mmap'd buffer or the file. BUT, the mmap'd buffer will get accessed again, but it will be new data. Maybe there might be a way to tell that the cache is dirty and needs to be refreshed or just forgotten, maybe that's what we're trying to do through the posix_fadvise. I have tried posix_madvise, but I receive errors from the call mostly I suspect because the mmap'd buffers are PCI addressable RAM from pci_alloc_consistent.

My first implementation is a simple write from my mmap'd buffer to the file through a write call.

write(fd,mmap_buffer[i],4*MB);

Or, I use O_DIRECT to open the file and then I create a page aligned buffer with posix_malign to 4MB and then I need to call...

memcpy(malign_buf,mmap_buffer[i],4*MB);
write(fd,malign_buf,4*MB);

I have not had any performace boosts by utilizing posix_fadvise or even O_DIRECT.


Thanks,

Matt