LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 01-17-2017, 06:21 AM   #1
Seongyun Ko
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Rep: Reputation: Disabled
How to avoid writing back the pages in mmaped-region to disks?


I use Cent OS 7 with kernel 3.10.0-327.36.1.el7.x86_64.

The kernel provides memory mapping capability via mmap() API.

In my C++ program, I use memory mapping for a file, then I update its contents as if it's an array.
I found that during the updates on the mapped memory regions the dirty pages are written back to disks incurring huge disk I/O, which I want to avoid.

I tried mlock-ing to lock the pages in memory. But it still writes back the dirty pages.
I guess the background kernel threads keep writing back the dirty pages.

Can I 'lock' a certain memory mapped region so that even the dirty pages are not written back to disks?
 
Old 01-17-2017, 09:40 AM   #2
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
When you map a file, Linux is going to write data back to that file quickly, because it knows that it is a file. It very-briefly buffers the data, then writes it, because, "that's what disk-files are supposed to do."

If you are incurring very high disk-I/O volumes ... which doesn't surprise me in the least ... then you need to change your algorithm; change your approach. Random-access disk I/O is very expensive, because it requires two mechanical movements: the read/write head must "seek," then the disk must rotate into position. "Milliseconds add up very fast."

One strategy might be to accumulate updates that you intend to make to the memory-mapped file, initially using a non-mapped buffer, then apply the updates to the memory-mapped file in ascending order by some kind of key that is relative to the position of the data in the file. In this way, the disk drive has less work to do.

More generally, instead of mapping the entire file and treating it like an array, set a window into that file and do all of the updates that fall within that window before moving the window. Accumulate all changes that are to be made to a single record before updating that record in the mapped file.

It is also possible to "go completely retro" and do it the way that COBOL programmers did it when they didn't have disk space: only magnetic tape. If you apply sorted updates to a sorted file, you produce another sorted version of that file. (Or, you update the file extremely efficiently.)

In a famous-to-me program that I wrote (and sold!) a decade ago, I made an algorithm run more than one hundred times faster by applying these techniques.

Last edited by sundialsvcs; 01-17-2017 at 09:41 AM.
 
  


Reply

Tags
mmap



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Writing to MAP_PRIVATE mmaped file rhw Linux - Newbie 1 11-01-2016 08:36 PM
LXer: Italian region asks for help to avoid software lock-in… to Microsoft LXer Syndicated Linux News 0 11-26-2010 05:10 AM
can I avoid writing to bad sectors on a CD-RW? newbiesforever General 5 02-06-2008 02:28 AM
should I avoid writing to Windows partition? newbiesforever Linux - General 11 08-23-2006 02:55 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 07:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration