LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 12-17-2006, 12:57 AM   #1
anamericanjoe
Member
 
Registered: May 2006
Posts: 69

Rep: Reputation: 16
mmap tutorial (C/C++)


I am trying to learn how to use mmap and am actively looking for a good tutorial, or a set of good examples that demonstrate mmap use.

I have read through Memory Mapped Files, but it only demonstrates mmap with a read example. I realize that this example recommends simply using PROT_WRITE to write to a file, however I have tried doing so and get a Segmentation Fault when I run my program.

Does anyone have any good examples of mmap that write to a file?
 
Old 12-18-2006, 08:51 AM   #2
_john_i_
Member
 
Registered: Aug 2003
Location: Austin, TX
Distribution: Linux from Scratch
Posts: 52

Rep: Reputation: 15
I'm not 100% sure, but I think if you write to a memory-mapped file the space must already exist in the file. (i.e., if you mmap 1 meg, the file must be at least 1 meg.) I don't think mmap will append bytes to the end of a file.
 
Old 12-18-2006, 11:01 AM   #3
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: Debian
Posts: 2,536

Rep: Reputation: 111Reputation: 111
Here's a commented working example for writing to a file through mmap():
Code:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>

#define FILEPATH "/tmp/mmapped.bin"
#define NUMINTS  (1000)
#define FILESIZE (NUMINTS * sizeof(int))

int main(int argc, char *argv[])
{
    int i;
    int fd;
    int result;
    int *map;  /* mmapped array of int's */

    /* Open a file for writing.
     *  - Creating the file if it doesn't exist.
     *  - Truncating it to 0 size if it already exists. (not really needed)
     *
     * Note: "O_WRONLY" mode is not sufficient when mmaping.
     */
    fd = open(FILEPATH, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);
    if (fd == -1) {
	perror("Error opening file for writing");
	exit(EXIT_FAILURE);
    }

    /* Stretch the file size to the size of the (mmapped) array of ints
     */
    result = lseek(fd, FILESIZE-1, SEEK_SET);
    if (result == -1) {
	close(fd);
	perror("Error calling lseek() to 'stretch' the file");
	exit(EXIT_FAILURE);
    }
    
    /* Something needs to be written at the end of the file to
     * have the file actually have the new size.
     * Just writing an empty string at the current file position will do.
     *
     * Note:
     *  - The current position in the file is at the end of the stretched 
     *    file due to the call to lseek().
     *  - An empty string is actually a single '\0' character, so a zero-byte
     *    will be written at the last byte of the file.
     */
    result = write(fd, "", 1);
    if (result != 1) {
	close(fd);
	perror("Error writing last byte of the file");
	exit(EXIT_FAILURE);
    }

    /* Now the file is ready to be mmapped.
     */
    map = mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (map == MAP_FAILED) {
	close(fd);
	perror("Error mmapping the file");
	exit(EXIT_FAILURE);
    }
    
    /* Now write int's to the file as if it were memory (an array of ints).
     */
    for (i = 1; i <=NUMINTS; ++i) {
	map[i] = 2 * i; 
    }

    /* Don't forget to free the mmapped memory
     */
    if (munmap(map, FILESIZE) == -1) {
	perror("Error un-mmapping the file");
	/* Decide here whether to close(fd) and exit() or not. Depends... */
    }

    /* Un-mmaping doesn't close the file, so we still need to do that.
     */
    close(fd);
    return 0;
}
Reading the file created by the example above, is simpler as we don't need to create a file and "stretch" it to the mmapped size. So just remove that part, and open the file for reading only, and remove the PROT_WRITE flag from the mmap call.

Code:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>

#define FILEPATH "/tmp/mmapped.bin"
#define NUMINTS  (1000)
#define FILESIZE (NUMINTS * sizeof(int))

int main(int argc, char *argv[])
{
    int i;
    int fd;
    int *map;  /* mmapped array of int's */

    fd = open(FILEPATH, O_RDONLY);
    if (fd == -1) {
	perror("Error opening file for reading");
	exit(EXIT_FAILURE);
    }

    map = mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);
    if (map == MAP_FAILED) {
	close(fd);
	perror("Error mmapping the file");
	exit(EXIT_FAILURE);
    }
    
    /* Read the file int-by-int from the mmap
     */
    for (i = 1; i <=NUMINTS; ++i) {
	printf("%d: %d\n", i, map[i]);
    }

    if (munmap(map, FILESIZE) == -1) {
	perror("Error un-mmapping the file");
    }
    close(fd);
    return 0;
}
Hope this helps.
 
Old 12-18-2006, 10:50 PM   #4
crabboy
Senior Member
 
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,821

Rep: Reputation: 121Reputation: 121
The forum rules do not permit advertising. Please visit http://www.linuxquestions.org/advertising/ for more information on advertising. Feel free to contact the forum admin if you have any questions about this policy.
 
Old 02-17-2007, 06:38 AM   #5
traene
Member
 
Registered: Jan 2005
Distribution: Archlinux, Debian, Centos
Posts: 222

Rep: Reputation: 35
I tried the example programms. Note that you should add a (int*) before the mmap calls, like:
Code:
map = (int*)mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);
 
Old 07-25-2007, 05:05 AM   #6
Ifing
LQ Newbie
 
Registered: Jul 2007
Posts: 2

Rep: Reputation: 0
Thanks for the example.

When using PROT_READ, is there an advantage to using mmap()
rather than "normally" reading the file into memory, e.g:
<code>
fread(map, sizeof(int), 1000, fd);
</code>

What I am looking for is a way to use mmap() to create a "virtual" memory map
for the whole file, without actually reading it into memory.
As I understand it, mmap() can not do this?
 
Old 07-25-2007, 12:28 PM   #7
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: Debian
Posts: 2,536

Rep: Reputation: 111Reputation: 111
Quote:
Originally Posted by Ifing
What I am looking for is a way to use mmap() to create a "virtual" memory map for the whole file, without actually reading it into memory.
As I understand it, mmap() can not do this?
As I understand mmap(), that is exactly what mmap does: "create a "virtual" memory map for the whole file, without actually reading it into memory".

If mmap() does not do that, what would?
 
1 members found this post helpful.
Old 07-26-2007, 03:13 AM   #8
Ifing
LQ Newbie
 
Registered: Jul 2007
Posts: 2

Rep: Reputation: 0
You are right - it does create a virtual map, rather than "mirroring" the file in RAM as I first thought.
 
1 members found this post helpful.
Old 10-21-2007, 09:49 AM   #9
hexingu2
LQ Newbie
 
Registered: Sep 2004
Posts: 2

Rep: Reputation: 0
Proper way to end file -EOF?

Hi to all:

I was wondering, if I wanted to put some kind of marker at the end of a memmapped file so as to have something to mark the end of the file, what is the proper way to do this?

For instance, I want to have a routine that will write out a variable amount of, say, ints to a file for whatever reason. This routine will create a new file or truncate an existing file, then write out however many ints are waiting to be written, then put a mark at the end of the file. A read routine will then later come back, remap the file, and read in ints until it finds the marker.

I've thought about using an EOF marker for this, but this equates to a "-1" in decimal. If any of the ints written out to the file happen to be a "-1", the read routine will think that this is the EOF marker and happily stop reading.

I know that one way to accomplish this is to build into the write routine to count the amount of data going out, then write that to the start of the file, or create a struct that will hold a count of the data separately from the data, but I'm looking for a way to do so as I've described above.

Any advice would be helpful.

Thanks!

hex
 
Old 10-24-2007, 02:33 PM   #10
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: Debian
Posts: 2,536

Rep: Reputation: 111Reputation: 111
Quote:
Originally Posted by hexingu2 View Post
I was wondering, if I wanted to put some kind of marker at the end of a memmapped file so as to have something to mark the end of the file, what is the proper way to do this?

For instance, I want to have a routine that will write out a variable amount of, say, ints to a file for whatever reason. This routine will create a new file or truncate an existing file, then write out however many ints are waiting to be written, then put a mark at the end of the file. A read routine will then later come back, remap the file, and read in ints until it finds the marker.

I've thought about using an EOF marker for this, but this equates to a "-1" in decimal. If any of the ints written out to the file happen to be a "-1", the read routine will think that this is the EOF marker and happily stop reading.

I know that one way to accomplish this is to build into the write routine to count the amount of data going out, then write that to the start of the file, or create a struct that will hold a count of the data separately from the data, but I'm looking for a way to do so as I've described above.
What would you need and EOF-marker for when using mmap?

Using mmap, you get to map a file's bytes. All of them, or part (offset + n). So you know the number of bytes since you allocated them yourself. Then you do not need an EOF marker, and you don't need to store the number of bytes (or ints or whatever) at the start.

If the number of bytes may have changed, you can get the size of the file beforehand using stat(), then you know the number of bytes to mmap().

If you want to read until EOF, use the normal ways to read/write files, using syscalls or the stdio library functions.

Of course you could use some end-of-file marking byte if you want. For end-of-record purposes or so. But then you will need to make sure the byte marking end-of-whatever does not appear in the data... This can be very unhandy, especially if you want to read/write binary data. You may need to do some encoding/decoding to avoid having the value of the marker-byte appear in the data.
 
Old 10-24-2007, 09:53 PM   #11
hexingu2
LQ Newbie
 
Registered: Sep 2004
Posts: 2

Rep: Reputation: 0
HKO - Thank you. Stat'ing the file to find the size is exactly what I was looking for, but was obviously being too dense to think about it the correct way.

This is just some quick dirty sample code to prove that your suggestion works like I need (for determining how many arbitrary ints have previously been written to a data file):
---
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char * argv[]){
struct stat buffer;
int status;
int fd;
fd = open("/tmp/mmapped.bin", O_RDWR);
status = fstat(fd, &buffer);
printf("number of data items in file: %d\n", buffer.st_size/sizeof(int));
close(fd);
return 0;
}
---

So far, the concept has worked perfectly. Thanks again!

hex
 
Old 10-24-2007, 10:39 PM   #12
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,662
Blog Entries: 4

Rep: Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942Reputation: 3942
Stop and consider what mmap is designed to do: it grabs a section of the page and segment tables that define your program's virtual memory space, and arranges for memory reads-and-writes to that "region of memory" to be redirected to that file (instead of the usual virtual-memory swap file).

This is exactly the sort of mechanism that's used to implement "shared libraries," except that in this case the memory-segment allows both reads and writes.

Now, then ... "make of it what you will."
 
  


Reply

Tags
eof, mmap, tutorial



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
mmap culin Programming 5 12-13-2006 10:51 PM
mmap implementation ash_lly Programming 4 03-10-2006 01:16 AM
mmap() error FarAway Programming 3 03-30-2005 07:38 AM
using mmap AngryLlama Programming 1 02-09-2005 08:53 AM
mmap problem os2 Programming 4 06-21-2004 05:24 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:10 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration