LinuxQuestions.org - mmap tutorial (C/C++)

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - mmap tutorial (C/C++) (https://www.linuxquestions.org/questions/programming-9/mmap-tutorial-c-c-511265/)

mmap tutorial (C/C++)

I am trying to learn how to use mmap and am actively looking for a good tutorial, or a set of good examples that demonstrate mmap use.

I have read through Memory Mapped Files, but it only demonstrates mmap with a read example. I realize that this example recommends simply using PROT_WRITE to write to a file, however I have tried doing so and get a Segmentation Fault when I run my program.

Does anyone have any good examples of mmap that write to a file?

I'm not 100% sure, but I think if you write to a memory-mapped file the space must already exist in the file. (i.e., if you mmap 1 meg, the file must be at least 1 meg.) I don't think mmap will append bytes to the end of a file.

Here's a commented working example for writing to a file through mmap():

Code:

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

#include <sys/stat.h>

#include <unistd.h>

#include <fcntl.h>

#include <sys/mman.h>



#define FILEPATH "/tmp/mmapped.bin"

#define NUMINTS  (1000)

#define FILESIZE (NUMINTS * sizeof(int))



int main(int argc, char *argv[])

{

    int i;

    int fd;

    int result;

    int *map;  /* mmapped array of int's */



    /* Open a file for writing.

    *  - Creating the file if it doesn't exist.

    *  - Truncating it to 0 size if it already exists. (not really needed)

    *

    * Note: "O_WRONLY" mode is not sufficient when mmaping.

    */

    fd = open(FILEPATH, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);

    if (fd == -1) {

        perror("Error opening file for writing");

        exit(EXIT_FAILURE);

    }



    /* Stretch the file size to the size of the (mmapped) array of ints

    */

    result = lseek(fd, FILESIZE-1, SEEK_SET);

    if (result == -1) {

        close(fd);

        perror("Error calling lseek() to 'stretch' the file");

        exit(EXIT_FAILURE);

    }

    

    /* Something needs to be written at the end of the file to

    * have the file actually have the new size.

    * Just writing an empty string at the current file position will do.

    *

    * Note:

    *  - The current position in the file is at the end of the stretched 

    *    file due to the call to lseek().

    *  - An empty string is actually a single '\0' character, so a zero-byte

    *    will be written at the last byte of the file.

    */

    result = write(fd, "", 1);

    if (result != 1) {

        close(fd);

        perror("Error writing last byte of the file");

        exit(EXIT_FAILURE);

    }



    /* Now the file is ready to be mmapped.

    */

    map = mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

    if (map == MAP_FAILED) {

        close(fd);

        perror("Error mmapping the file");

        exit(EXIT_FAILURE);

    }

    

    /* Now write int's to the file as if it were memory (an array of ints).

    */

    for (i = 1; i <=NUMINTS; ++i) {

        map[i] = 2 * i; 

    }



    /* Don't forget to free the mmapped memory

    */

    if (munmap(map, FILESIZE) == -1) {

        perror("Error un-mmapping the file");

        /* Decide here whether to close(fd) and exit() or not. Depends... */

    }



    /* Un-mmaping doesn't close the file, so we still need to do that.

    */

    close(fd);

    return 0;

}

Reading the file created by the example above, is simpler as we don't need to create a file and "stretch" it to the mmapped size. So just remove that part, and open the file for reading only, and remove the PROT_WRITE flag from the mmap call.

Code:

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

#include <sys/stat.h>

#include <unistd.h>

#include <fcntl.h>

#include <sys/mman.h>



#define FILEPATH "/tmp/mmapped.bin"

#define NUMINTS  (1000)

#define FILESIZE (NUMINTS * sizeof(int))



int main(int argc, char *argv[])

{

    int i;

    int fd;

    int *map;  /* mmapped array of int's */



    fd = open(FILEPATH, O_RDONLY);

    if (fd == -1) {

        perror("Error opening file for reading");

        exit(EXIT_FAILURE);

    }



    map = mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);

    if (map == MAP_FAILED) {

        close(fd);

        perror("Error mmapping the file");

        exit(EXIT_FAILURE);

    }

    

    /* Read the file int-by-int from the mmap

    */

    for (i = 1; i <=NUMINTS; ++i) {

        printf("%d: %d\n", i, map[i]);

    }



    if (munmap(map, FILESIZE) == -1) {

        perror("Error un-mmapping the file");

    }

    close(fd);

    return 0;

}

Hope this helps.

The forum rules do not permit advertising. Please visit http://www.linuxquestions.org/advertising/ for more information on advertising. Feel free to contact the forum admin if you have any questions about this policy.

I tried the example programms. Note that you should add a (int*) before the mmap calls, like:

Code:

map = (int*)mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);

Thanks for the example.

When using PROT_READ, is there an advantage to using mmap()
rather than "normally" reading the file into memory, e.g:
<code>
fread(map, sizeof(int), 1000, fd);
</code>

What I am looking for is a way to use mmap() to create a "virtual" memory map
for the whole file, without actually reading it into memory.
As I understand it, mmap() can not do this?

Quote:

Originally Posted by Ifing

What I am looking for is a way to use mmap() to create a "virtual" memory map for the whole file, without actually reading it into memory.
As I understand it, mmap() can not do this?

As I understand mmap(), that is exactly what mmap does: "create a "virtual" memory map for the whole file, without actually reading it into memory".

If mmap() does not do that, what would?

You are right - it does create a virtual map, rather than "mirroring" the file in RAM as I first thought.

Proper way to end file -EOF?

Hi to all:

I was wondering, if I wanted to put some kind of marker at the end of a memmapped file so as to have something to mark the end of the file, what is the proper way to do this?

For instance, I want to have a routine that will write out a variable amount of, say, ints to a file for whatever reason. This routine will create a new file or truncate an existing file, then write out however many ints are waiting to be written, then put a mark at the end of the file. A read routine will then later come back, remap the file, and read in ints until it finds the marker.

I've thought about using an EOF marker for this, but this equates to a "-1" in decimal. If any of the ints written out to the file happen to be a "-1", the read routine will think that this is the EOF marker and happily stop reading.

I know that one way to accomplish this is to build into the write routine to count the amount of data going out, then write that to the start of the file, or create a struct that will hold a count of the data separately from the data, but I'm looking for a way to do so as I've described above.

Any advice would be helpful.

Thanks!

hex

Quote:

Originally Posted by hexingu2 (Post 2931616)

I was wondering, if I wanted to put some kind of marker at the end of a memmapped file so as to have something to mark the end of the file, what is the proper way to do this?

For instance, I want to have a routine that will write out a variable amount of, say, ints to a file for whatever reason. This routine will create a new file or truncate an existing file, then write out however many ints are waiting to be written, then put a mark at the end of the file. A read routine will then later come back, remap the file, and read in ints until it finds the marker.

I've thought about using an EOF marker for this, but this equates to a "-1" in decimal. If any of the ints written out to the file happen to be a "-1", the read routine will think that this is the EOF marker and happily stop reading.

I know that one way to accomplish this is to build into the write routine to count the amount of data going out, then write that to the start of the file, or create a struct that will hold a count of the data separately from the data, but I'm looking for a way to do so as I've described above.

What would you need and EOF-marker for when using mmap?

Using mmap, you get to map a file's bytes. All of them, or part (offset + n). So you know the number of bytes since you allocated them yourself. Then you do not need an EOF marker, and you don't need to store the number of bytes (or ints or whatever) at the start.

If the number of bytes may have changed, you can get the size of the file beforehand using stat(), then you know the number of bytes to mmap().

If you want to read until EOF, use the normal ways to read/write files, using syscalls or the stdio library functions.

Of course you could use some end-of-file marking byte if you want. For end-of-record purposes or so. But then you will need to make sure the byte marking end-of-whatever does not appear in the data... This can be very unhandy, especially if you want to read/write binary data. You may need to do some encoding/decoding to avoid having the value of the marker-byte appear in the data.

HKO - Thank you. Stat'ing the file to find the size is exactly what I was looking for, but was obviously being too dense to think about it the correct way.

This is just some quick dirty sample code to prove that your suggestion works like I need (for determining how many arbitrary ints have previously been written to a data file):
---
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char * argv[]){
struct stat buffer;
int status;
int fd;
fd = open("/tmp/mmapped.bin", O_RDWR);
status = fstat(fd, &buffer);
printf("number of data items in file: %d\n", buffer.st_size/sizeof(int));
close(fd);
return 0;
}
---

So far, the concept has worked perfectly. Thanks again!

hex

Stop and consider what mmap is designed to do: it grabs a section of the page and segment tables that define your program's virtual memory space, and arranges for memory reads-and-writes to that "region of memory" to be redirected to that file (instead of the usual virtual-memory swap file).

This is exactly the sort of mechanism that's used to implement "shared libraries," except that in this case the memory-segment allows both reads and writes.

Now, then ... "make of it what you will."