I have RHAS 4 kernel 2.6.9-55.ELsmp, install in a dell server PE2950 with 16 Gb of ram.

Here's the problem, when i start oracle instances, the server start using the memory around 4Gb, then start tar with gzip option to backup some file on local disk. If i stop the oracle instances, it seems that the memory is not release, i used top/free command to check. So i start back the oracle instances again, it will start using the available free memory up to around 12GB. So when the oracle will continue to use all the free available instances and then starting the use the swap space.

But when there is no free available memory, my oracle start to have problem like below :

Cause: AFPBAN failed due to ORA-03114: not connected to ORACLE . The SQL
statement being executed at the time of the error was: and was executed
from the file . ORACLE error 3114 in AFPRSR-Resubmit_Time Cause:
AFPRSR-Resubmit_Time failed due to ORA-03114: not connected to ORACLE

So if i used the kernel hugemem, will it improve the large memory usage in my case ? I also notice if tar with gzip not running, the memory usage will also drop. Is that have any related to kernel bugs ? Anyone, kindly please advice, thanks...

Well, it might, but I'd contact Oracle support first. There is a whole list of things you have to set up first, before installing Oracle, and alot of them have to do with the kernel and memory. If they're not set right, Oracle will not behave correctly.

You have a supported configuration with RH and Oracle, so I'd contact Oracle first, and see what they say.