If you had done a forum search for SHMMAX you would already have your answer. Among other threads you would have found this thread where the OP found the answer despite my well intentioned but off target advice:
In that thread, jlightner suggest this command to see the current settings and a couple methods of adjusting them were discussed:
sysctl -a |grep -i shm
Of course it is possible those settings are OK and the problem is lack of swap space and/or a problem in commit limits. Assuming it is a 32 bit system, you might even be running into some application or kernel virtual memory limit. So if the answers in that or similar existing threads aren't right for your situation, post more info (such as the name of the failing application and output from the free command). Someone will give you more specific advice for your situation.