[SOLVED] Is there a system call other than 'open' for opening very large files?

dariyoosh · 08-15-2009, 03:48 PM

Dear all,

Inside a C program, I want to open a very big file (about 12 GB) in order to read its
content. Here is the code:

Code:

    /*
          argv[1] contains the path to the file.
    */
    inputFileDescriptor = open(argv[1], O_RDONLY);
    if (inputFileDescriptor < 0)
    {
        fprintf(stderr, "Error: Could not open the input file\n");
        fprintf(stderr, "%s\n", strerror(errno));
        return 1;
    }

And it seems that the file is too big, because I receive the following error message:

Code:

Error: Could not open the input file
Value too large for defined data type

Is there any other system call, allowing to open very big files?

Thanks in advance,
Dariyoosh

dmail · 08-15-2009, 04:41 PM

Add O_LARGEFILE to the open flags or use open64 which uses this internally. IIRC there is a posix define which you can use _LARGEFILE_SOURCE.
You would probably be advised to mmap parts of the file at a time.

dariyoosh · 08-16-2009, 06:33 AM

Quote:

Originally Posted by dmail

Add O_LARGEFILE to the open flags or use open64 which uses this internally. IIRC there is a posix define which you can use _LARGEFILE_SOURCE.
You would probably be advised to mmap parts of the file at a time.

Hello there,

Thank you very much for your help. I'll try this method.

Kind Regards,
Dariyoosh

bigearsbilly · 08-16-2009, 04:29 PM

mmap is another way top open files
but...
that is a very big file.

dmail · 08-16-2009, 05:57 PM

Quote:

Originally Posted by bigearsbilly

mmap is another way top open files
but...
that is a very big file.

But mmap requires a file descriptor you still need to open the file first no?

ta0kira · 08-16-2009, 07:13 PM

You need the O_LARGEFILE flag with open:

Code:

#define _LARGEFILE64_SOURCE
#include <sys/stat.h>
//...

    /*
          argv[1] contains the path to the file.
    */
    inputFileDescriptor = open(argv[1], O_RDONLY | O_LARGEFILE);
    if (inputFileDescriptor < 0)
    {
        fprintf(stderr, "Error: Could not open the input file\n");
        fprintf(stderr, "%s\n", strerror(errno));
        return 1;
    }

Kevin Barry

dariyoosh · 08-17-2009, 05:09 AM

Quote:

Originally Posted by ta0kira

You need the O_LARGEFILE flag with open:

Code:

#define _LARGEFILE64_SOURCE
#include <sys/stat.h>
//...

    /*
          argv[1] contains the path to the file.
    */
    inputFileDescriptor = open(argv[1], O_RDONLY | O_LARGEFILE);
    if (inputFileDescriptor < 0)
    {
        fprintf(stderr, "Error: Could not open the input file\n");
        fprintf(stderr, "%s\n", strerror(errno));
        return 1;
    }

Kevin Barry

Hello there,

Thanks a lot for your solution. I also applied this method and it worked pretty well.

I thank all of you for your attention to my problem and for your help.

Kind Regards,
Dariyoosh