[C/C++] large files (>4gb) and 32bits machines

asimov · 12-06-2007, 12:20 PM

Hi,

My program needs to handle very large binary files (bigger than 4GB), and I can't find a way to open those files for reading / writing on my old 32bit laptop.

when trying to use FILE* h=fopen64(filename, "rb"), i get only 4 bytes pointer.

it there some sort of FILE64*, or a way to overcome this issues on 32 bits machines?

orgcandman · 12-06-2007, 12:35 PM

Quote:

Originally Posted by asimov

Hi,

My program needs to handle very large binary files (bigger than 4GB), and I can't find a way to open those files for reading / writing on my old 32bit laptop.

when trying to use FILE* h=fopen64(filename, "rb"), i get only 4 bytes pointer.

it there some sort of FILE64*, or a way to overcome this issues on 32 bits machines?

I'm fighting a fire with pressure at 4 gals / second. I've changed my pressure gauge to 8 gals / second. Why didn't I get a new hose?

analogy aside, the file pointer is just a descriptor that carries stream information. The thing that overcomes the 32-bit limitation is buried deep within the read/seek/write api that is used with that file descriptor (I think there are separate fread64 and fwrite64 functions that exist, but I could be wrong here).

-Aaron

asimov · 12-06-2007, 01:00 PM

nice analogy

I thought the file handle/descriptor is a pointer to a position on the file. I'll look into fread64 and fwrite64.

thank you for your answer.

BTW, sometimes overpressure might explode the hose, then you'll really need a new one

schneidz · 12-06-2007, 02:19 PM

by and by, fgetc() seems to play nicely with fopen64().

asimov · 12-06-2007, 02:55 PM

isn't fread64() more efficient then fgetc() for reading more than one character?

i need to read only portion of files. with performance as top priority, is there any more efficient way then using fopen64, fsetpos64/fseek, fread64/memcpy, fclose sequences?

asimov · 12-07-2007, 05:39 AM

and another question,

On windows, you can speed up reading from file by using sector-aligned reading (reading full sectors from disk). how that can be achieved with linux?

what is the equivalent of windows's GetDiskFreeSpace(), in order find out how much bytes i have on my HD per sector?

matthewg42 · 12-07-2007, 06:07 AM

This might be useful information about handling large files:

https://www.linuxquestions.org/quest...3/#post2951079

bigearsbilly · 12-07-2007, 09:11 AM

you can speed up reading using the mmap system call.
which reads a file directly into memory.

it's easy too.

I've mmap'd a 160Meg file recently.

4 gig on a laptop should be fun