Well, malloc's argument is generally a 4 byte integer, so you're definitely not getting 10 gigs out of it : ) The size of your virtual memory is (probably) only 4 gigs, and as mentioned some of that is reserved for kernel memory. Then some more is taken by your app's code and any shared libraries you're using.
Here's a little trick to figure out what values malloc will barf on:
Code:
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <inttypes.h>
int main()
{
size_t min = 1;
size_t max = ULONG_MAX;
while (1) {
void *p;
size_t middle;
if (min >= max)
break;
middle = (size_t) ((((uint64_t)min) + ((uint64_t)max)) / 2LL);
p = malloc(middle);
free(p);
if (p == NULL)
max = middle - 1;
else
min = middle + 1;
}
printf("%u = %x\n", min, min);
return 0;
}
On my system, I can get around 2,500,000,000 bytes.
However, memory management on linux is optimistic, so it's possible that you can get a non-null value from malloc and then have the OS start shooting processes when you actually go to use the memory you got. That's called the OOM killer.
Realistically, if you need gigs of memory to do whatever computation you're doing, you should figure out a way to write parts you aren't using to disk.