How much memory can all the processes allocate all together?
 

I wrote the following program and ran 3 instances of the same program concurrently.
================================
#include<stdio.h>
#include<stdlib.h>
int main()
{
unsigned long int i =0;
while(1)
{

if(malloc(100000) == NULL)
{
printf("Failed \n");
return 0;
}
i++;

}// end while
}// end main

=================================

When I ran the program, each instances terminated as
Failed 32031
Failed 32022
Failed 32027

Which means each instances can allocate around 3GB of memory successfully at the same time, which is totalling to 9GB.

But the actual RAM is 512 and swap is 1GB.

How is this possible?
From where could it allocate 9GB, when the available space is 1.5 GB( RAM + SWAP)?

The kernel does lots to magic to protect you from yourself.
Try touching the memory and see what happens.