Hello amal_s, welcome to LQ,
as far as I know (I'm not an expert for hardware) the limitations are 3GB per process on x86-Linux with pae-kernel (the pae-kernel supports up to 64GB of RAM on a 32bit system, but limits each process to a maximum of 3GB of RAM).
On a 64bit there is no 3GB limit per process. This is true for Linux in general, but I don't know if and which possibilities a programmer has, to limit the memory for one process of his application.
You should use the report-button at your post and ask a Moderator to move your thread to the "Programming"-section of LQ here http://www.linuxquestions.org/questions/programming-9/
. I suppose there are people who are capable to answer your question in more detail.