Hi,
I have a generic doubt of swap space.
Once a process has been swapped out, will this actually be removed from swap other than when the process is killed ? Bcoz I read somewhere that even if the process is swapped in, the system still maintains it in swap to avoid the swapin swapouts in future. We have a server where the swap usage reaches almost 100% in 10 days after system is rebooted. But swapin-swapout is very less after this.

Can you give hardware info about your server?

Maybe the size of RAM is not sufficient.

Tasks haven't been "swapped out" since the Linux 2 era - only annonomous memory is subject to swap.Eventually. Running the page-table queues is expensive if not required, so it is deferred where possible. If the pages are needed for another task, they are released immediately.Same comment applies - if required, they are released immediately. Likewise if the page in memory is altered.Maybe you have an application that acquires memory until it can get no more, then proceeds with that. Monitoring usage over time might give some hints.

It may be possible to trace what is causing swap to fill up. The system ought to manage it but depending on how old the programs are they may simply try to use swap. A few thousand copies of some program may fill it.

I might be tempted to see what https://en.wikipedia.org/wiki/Swappiness changes may do.

Kind of suspect one of two issues. One that you do really need more swap or two that some memory leak causing it. Guess along that way some program is misbehaving or being caused to misbehave.

For another thread I was playing with "top" yesterday - it's easy to add a "Swap" column to see what is currently allocated. Perhaps set up a batch run after a boot with say a 10-20 minute interval to write to a file. That way you can see the increase occurring per task.

Hi,
Its a long reply

System had 6G ram initially when the issue was reported.
Swap was reaching almost 100% in 1 to 2 weeks time. System still had 1G buffer/cache.
The problem here is that there is no single process occupying high memory.
If you run top and sort based on Memory or swap, you will find 5-6 processes using more than 10-15% of the total memory/swap.
This rules out the possibility of memory leak.
And another interesting thing is that this issue happens for multiple nodes in 2 different clusters (running almost same list of applications.)
This was a clear sign that system was running out of memory and swap.
So we increased memory from 6G to 8G hoping that this would solve the issue.
But it didn't.
swap memory usage still rose to more than 90% in 10 days.
The buffer/cache memory which previously used to be 1G, now became 3G
(The behavior was not same in all the nodes. For some nodes it didn't even go beyond 50%).

The only logical justification I could find is that system really needed more swap memory.
Bcoz what I understand is that processes/pages which were swapped out once, were never removed from swap memory.
Unless you manually clear it or the application is killed/stopped.
But somewhere I read that even if a process is killed, system won't really show this space as free bcoz linux uses Lazy memory management as removing page table entries frequently is an expensive process. Until some apps request for more memory.

Any more inputs appreciated.