Hi friends and Gurus,

am facing a issue with one of my server, my problem is linux rhel5 Dell server is having 4GB of physical memory and 8gb of swap.when i got a ticket for swap using more space. then i checked, out of 8gb of swap 7gb is consumed and physical memory 95% free.

here my query is as per my knowledge when memory getting over loaded it will take the help of swap. in my case when memory is consuming only 5% and swap is 90% how it is possible....is there any victim like meee.

any piece of help is appreciated.


Thanks In Advance
JNREDDY.

Hi jnreddy,

First of all if you have got RHEL5 installed then you should have also got support for it. Right? So, I would suggest you to get in touch with Redhat.

Yes, your concept of swap usage is correct but the configuration itself it wrong. You shouldn't be assigning 8 GB to swap partition because you are working on linux not on Windows :-). If your system is using swap instead of physical memory then there is something wrong with your system as it shouldn't go to swap when physical memory is available.

Is your system 32 bit or 64 bit? If it is a 32 bit system and you have got 4 GB of memory then the server won't even look at swap unless required. Which kernel are you using? Normal one or kernel with PAE.

The amount of swap you need for a system is solely dependent on the workload of that machine, not the OS you are running.

Wrong. It is normal for a Linux system to swap out stale memory pages to get more cache and so speed up the system. Even on a system with a large amount of free memory it is normal to have some of your swap being used, nothing wrong with it.

May I ask where you got that info from? Why shouldn't a 32 bit system with 4GB don't use swap?

Exactly, that is why you pay for the support.

As far as I am aware if you think that system will work under heavy load then it is better to put physical RAM rather than to have large swap because ultimately you are compromising with the performance. Doesn't matter whether it is linux or windows, for system to read from swap has to make disk I/O request which involves more time than quering RAM.

It is abnormal for any system (whether windows or linux) to use swap memory when physical memory is free. 90% physical memory usage and 10% swap usage is acceptable but 10% physical memory usage and 90% swap usage not acceptable.

As far as I am aware 32 bit system can address 2 to the power 32 memory locations which I think 4 GB is good enough for. I did say it won't even look for swap unless really required.

Agree on that!

1 members found this post helpful.

Depending on what exactly you mean by "free", that is either strange or extremely strange.

Depending on what is running and has run recently on that system, those numbers could represent perfectly normal operation and no problem. But more likely, they indicate some significant problem, such as a memory leak.

It is perfectly normal for stale anonymous pages to be written out to swap while more recent pages are kept in cache. Some tools may report cache memory as "free", so if "free" includes cache, the question is why such a large fraction of pages are stale anonymous. The typical reason would be a memory leak, which should be investigated by seeing which processes own all that stale anonymous memory.

If "free" does not include cache, then something must have used and released that memory to leave it free. Then you looked at memory stats before the swapped out pages were touched again. Assuming you didn't time your check to a narrow moment and see something not normally present, that begs the question of what big processes you are running that don't need most of their memory back more quickly. Regardless of that answer, there is also what big process used and released all that ram? If that process cannot be tamed, your system is seriously under configured.

Wrong.
Wrong.
Wrong.
It is normal to use swap for stale anonymous pages. Only these extreme amounts are unlikely to represent correct operation. Even that is just "unlikely", not impossible to be correct operation.
Wrong. 32 bit vs. 64 bit is irrelevant to this.
Still wrong.
That is close to correct. 7GB of swap use with very little physical ram use is unlikely to come from correct operation. Likely some problem should be found and fixed. But that takes deeper investigation. The relationships are not nearly as clear cut as your statements assume.

32 bits can address 3GB for EACH process, plus some for the kernel, plus cache, plus a few other things. So with several processes the total can go way above 4GB.

I hope that works. I don't have a lot of experience with Red Hat support. Most software vendor support is worthless even in clear cut cases where there is no possible ambiguity about the problem being the fault of the vendor. In a case like this, I think it is more likely to be some kind of misuse rather than anything that should be considered Red Hat's responsibility. If they choose to diagnose it for you, great. They don't charge enough to routinely diagnose such things for large numbers of clients.
The software vendor I work for provides extraordinarily good support and that certainly extends into diagnosing misuse and other issues that are entirely the customer's "fault" (not a word actually used, of course). But that is for a specialized product that costs way more per seat than Red Hat. Ordinary vendors' software support generally assumes the customer can rule out most possible misuse aspects of the problem before asking any hard questions.

1 members found this post helpful.

I may be wrong but here are the things that I would really like to know:

1. When system use more SWAP than RAM?
2. Why we need big SWAP? Any specific reason? If system is clogging both SWAP and RAM isn't it a sign that we need more physical RAM?

http://www.cyberciti.biz/tips/linux-swap-space.html

https://www.linux.com/news/software/...nux-swap-space

Acording to OP:

As per my understanding system put those things in swap which it will not going to use frequently or when system feels like it does not have enough RAM to hold the things. Am I right? If yes, then OP concept of system putting things in swap has to be right as well.

Agree. But does it still hold true when you have got free physical memory?

Could you please point me to a link that explains maximum memory that a 32 bit system can handle. Does this work for normal kernel without PAE.

As far as I know a 32 bit system with more than 4 GB of RAM use RAM > 4GB as swap. Doesn't make a sense to me putting more RAM if system will use it as SWAP.

This is what I was talking about:

1. 32 bit system: http://en.wikipedia.org/wiki/32-bit

2. 64 bit system: http://en.wikipedia.org/wiki/64-bit

The above URL tells us about the addressable memory for 32 bit and 64 bit systems. That is the reason I asked OP about system architecture.

That depends on the definition of "free". Some tools include cache memory as "free", because for most purposes cache memory is free. The system may swap out stale anonymous pages while most physical memory is held by the cache.

Alternately, memory might have been used by a process that is no longer present. When that memory is freed, the kernel does not read back pages from swap until those pages (or at least nearby ones) are accessed again.

I guess I didn't think of the right google search string. I tried
+"3GB per process" + linux
That should have found some detailed explanations of why 32 bit linux has that limit (which I understand myself, but don't want to type here). It found many posts in several forums that mention the limit. But I didn't see any good explanation.

PAE affects only the amount of physical memory Linux can use, not the amounts of virtual memory nor swap. Without PAE you can only have three and a fraction GB of physical memory, but each process can use 3GB of virtual memory, and the total of many processes can use far more than 4GB of swap.

Sorry, wrong again. With PAE, you can use memory up to 16GB equally as well as the first 3 and a fraction GB. PAE gives the ability to address 64GB physical ram, but the limit on the kernel's own virtual address space makes that impractical.

1 members found this post helpful.

Alright. I did further digging on this topic and found that a 32 bit system without PAE can only address 4 GB of RAM. Recommended swap space for a 4 GB system minimum is 2 GB. You can assign more SWAP space but system will use it as disk locations (PAE not required).

When I ran free command on my system it showed me free mem excluding cached. I think that is what OP meant when he said free mem.

I thank each and everyone my seniors and gurus, thanks for digging this issue, we are having support with Redhat i can log a case with redhat and request RCA for this issue but it would be my last option, my collegues are digging for this but unable to find the fix, now its above 40 hours its eating swap, he my luck is its a development box otherwise i was fired like any thing..

is there any option or any command to divert the swap utilization to physical memory....
"once again thanks a lot to spending your precious time for queries"


thanks
JNREDDY

You're welcome.

Paste the output of the following commands:

Provide the following information:

1. Applications running on this server.
2. Since when you started facing this issue.
3. Any chages made to this server recently.
4. Was it working fine initially.
5. Is 8 GB swap on single partition or scattered. Paste the output of the following:

As far as I know, there is no direct way to do something like this.

Now this thread has wandered around a bit, via some irrelevant stuff, but:

It would be advisable if the OP could post some hard information; there have been requests, eg,

You need to say something about the processes that you are running (...there may be something known for memory leaks...perhaps it is a long shot, but it shouldn't be excluded...). What processes seem to be using memory?

When you got the information about 'free' and 'physical memory' what tool did you use to get that information? Post, within code tags, the command that you used and the output, then we'll know.

Additionally
Am I right in thinking that you mean that you

• received a fault ticket
• the fault ticket specified that there was some kind of performance problem
• the performance problem is the thing that needs to be fixed, and that you have concluded that the strange memory usage profile is key to solving the performance problem (which may or may not be true...they probably are connected, but it is possibly not the case that trying to change the odd usage profile is the best way to attack the performance problem, assuming that there is a performance problem, of course)

Please also post the output of
swapon -s
vmstat 2 20

(the latter while the system is having the problem, if there is a particular time at which the problem is evident)

I thank you Salasi and T3RM1NVTOR and also who contributed their valuable time for this post.

I identified that some weblogic 3 instances are are running and consuming more swap space,tomorrow i would come up with the outputs,
today i got the same alert on other is also the same case, swap consumed more space...

Thanks and Regards
JNREDDY.