I see there is mallocoptions=disclaim that can be set on AIX for an ID. Then unused memory will be returned to the server from any processes that runs on that ID.

Is there a similar setting on Linux. On our Linux server, the process memory usage keeps growing. This is because,eventhough there is memory available from previous run of the program. the program expects contiguous memory blocks which is not available hence it goes back to the physical memory and asks for more. This results in memory growing for ever slowly.

On AIX when this mallocoptions=disclaim is set, unused memory is returned after the application stops and memory is got back form physical memory each time. I know it is a performance hit but it is better than a holding memory for nothing.

If a process is terminated, all of its memory will be reclaimed by the OS. If your system continuously asks for more memory, it might have a memory leak. You can use a tool called Valgrind to check for this.

The way the application runs is, the process is always running. It's multi threaded. The threads use memory and returns back
To process. Next time the thread processes, it requires memory but the memory returned is not good because it is fragmented. It needs contiguous memory. Hence the process gets more memory. This happens over and over again causing memory spike. If there was a way the process could return memory that wud be great.

You can't.

If memory is fragmented as described, it can't be deallocated... If it is not fragmented, you will get a chunk of the desired size.

The memory management already combines freed memory blocks (making a larger block) IF they are adjacent.