I'm trying to use O_DIRECT to open and read some very large files. I'm trying to read 2-30 GB files into memory and I don't need the caching facilities of the OS since I plan to fill the entire memory space.
The man page for open is a little sparse on details (man 2 open, man 2 read) on the maximum size I can use to read using the O_DIRECT flag when opening a file:
Code:
int in_fd = open(filename, O_RDONLY | O_DIRECT | O_LARGEFILE);
// Get the file size
struct stat64 statbuf;
fstat64(in_fd, &statbuf);
// Allocate memory for the file and align it to 512 bytes.
int d_mem=512;
char* idata = (char*)malloc(statbuf.st_size+d_mem);
size_t addr = (size_t)idata;
size_t offset = addr % d_mem;
if (offset)
addr += d_mem - offset;
char* data = (char*)addr;
// What I need to know is how to determine what maxReadSize is
size_t maxReadSize = 0x00800000;
size_t remaining = statbuf.st_size;
size_t total = 0;
while (remaining > 0) {
size_t toRead = remaining>maxReadSize?maxReadSize:remaining;
ssize_t result = read(in_fd, data + total, maxReadSize);
if (result <= 0) {
// Do error stuff ....
return 1;
}
total += toRead;
remaining -= toRead;
}
if (total != statbuf.st_size) {
fprintf(stderr, "Didn't read the right number of things\n");
// Do error stuff ....
return 1;
}
close(in_fd);
What I need to know is how to determine what the maxReadSize I can feed to read. There doesn't seem to be as many restrictions for cases not using O_DIRECT.
Thanks,
James
