Please Help Me! How to use more 2GB (>7GB) in a C program

minhsangniit · 03-04-2006, 01:02 AM

I have an C program (GNU C) and i need it use more 2GB (7-8GB) of RAM, and i used malloc to allocate >2GB of RAM and i got crash (when running - not compiling)- "Segmentation fault". What does option of GNU C compiler support it?

paulsm4 · 03-04-2006, 01:27 AM

Try looking at mmap.

But I'm not at all sure you're going to get 7GB of contiguous memory with *any* technique.

BTW: are you running a 64-bit OS?

minhsangniit · 03-04-2006, 01:32 AM

I'm running a 64-bit OS (Debian Linux 2.6.8), DUAL XEON 3.6GHz, 8GB RAM

jlliagre · 03-04-2006, 03:38 AM

Are you asking gcc to compile a 64 bit binary ?

Do you really want to use 7 GB of RAM, which is normally not a requirement for user space programs, but instead want to use 7 GB of virtual memory, and have the system mapping it to RAM when available ?

aluser · 03-04-2006, 02:50 PM

what do you get when you printf("%d\n", sizeof(size_t)) ?

(Since malloc() takes a size_t as its argument, it would be bad for you if it's 4 bytes..)

But it really does seem that you should be using mmap. I'm fairly certain I remember there being an mmap64 syscall, but I don't know if regular mmap calls that when you compile for 64bit or not.

primo · 03-05-2006, 03:04 AM

Quote:

Originally Posted by aluser

what do you get when you printf("%d\n", sizeof(size_t)) ?

(Since malloc() takes a size_t as its argument, it would be bad for you if it's 4 bytes..)

But it really does seem that you should be using mmap. I'm fairly certain I remember there being an mmap64 syscall, but I don't know if regular mmap calls that when you compile for 64bit or not.

No, mmap64() only offers an off64_t: support for mapping any portion of a large file like you would do with mmap()

It's not easy to use portions of memory larger than SIZE_MAX, but the real limit is SSIZE_MAX. Both read() and write() say they accept size_t but they return ssize_t so there are implementations that don't support sizes between SSIZE_MAX and SIZE_MAX - 2, and in those that do, code such as "if (read(fd, buf, size) < 0)" will fail horribly because SSIZE_MAX + 1 becomes negative on a signed variable.

I think you'd have to code a library on top of mmap and I/O calls and you'd have to keep an array of addresses and their lengths associated to mmap() returned pointers. But first try to see if your OS supports many of these.

Intimidator · 03-05-2006, 01:19 PM

why don't u try increasing the amount of swap.
swap space can be created on a separate partition or by creating a file on an existing partition and mounting it as swap

to create a 1 MB swap file:

Code:

 dd if=/dev/zero of=swapfile bs=1000000

 mkswap swapfile (This cmd makes the file suitable for swap)

 swapon swapfile (Turn on the swap usage)

Post your result

jlliagre · 03-05-2006, 02:34 PM

Quote:

Originally Posted by Intimidator

why don't u try increasing the amount of swap.
...

The OP has already 8 GB of RAM on his machine, there's likely no need to add swap for his test.

aluser · 03-05-2006, 08:23 PM

One hacky way to get it to work is use mmap with 2gb segments and MAP_FIXED to line them up against eachother. As long as you start by finding an 8gb area of your address space that's free (shouldn't be hard with a 64bit space; use pmap if unsure), there's not much to pontificate about.

minhsangniit · 03-06-2006, 08:06 PM

I think I can use "shared memory" to resolve my problem (because an application can get 2GB of RAM) and i will get the memory area's address to use in other application.

paulsm4 · 03-06-2006, 08:32 PM

Please remember:

* "memory" is memory: someplace to hold data.

* "shared memory", on the other hand, is an IPC.

Shared memory is a form of "InterProcess Communication"

Shared memory is for "sharing" data (not just "having" data).

* So don't use shared memory unless you genuinely need to communicate data between different processes.

* And, if you do, don't use shared memory unless you *synchronize* access to that shared data (e.g. with SysV semaphores)

IMHO .. PSM