LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 03-15-2017, 06:44 PM   #1
newtonbear
LQ Newbie
 
Registered: Mar 2017
Posts: 4

Rep: Reputation: Disabled
share memory page swapping out


Our distributed server runs for sometimes.
We are setting /proc/sys/vm/swappiness as 0.
Our service is based on JVM with memory mapped file with java MappedByteBuffer through RandomAccessFile which in top command shown as shared memory.
Recently we find that our service some of shared memory got flushing out from physical memory which causes some latency issue.
Just wonder is there any condition that memory mapped file still can be replaced by os?
Does anybody know the reason for this?

And if this issue happens, if we just restart our service, this issue will come up in next several hours. The only way we find to resolve it is to drop the file cache by echo 1 > /proc/sys/vm/drop_caches.
 
Old 03-15-2017, 11:44 PM   #2
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
Hi Newton,

Welcome to Linuxquestions

Quote:
Does anybody know the reason for this?
The condition you're describing is a form of "thrashing", the processor is consuming more cycles moving data in and out of shmem than it is processing your workload. The most common fix is to limit your java heap at instantiation .

Quote:
The only way we find to resolve it is to drop the file cache by echo 1 > /proc/sys/vm/drop_caches.
If you can dump your caches into /dev/null without suffering process failure of one kind or another, you should definitely limit the heap at instantiation.
java -Xms<initial heap size> -Xmx<maximum heap size> javaApp in a bash whack is normally how it's done.

Last edited by dijetlo; 03-15-2017 at 11:52 PM.
 
1 members found this post helpful.
Old 03-16-2017, 01:01 PM   #3
newtonbear
LQ Newbie
 
Registered: Mar 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Hi dijetlo
Thanks for kindly replying.
I understands it is a kind of thrashing.
I am more curiously about how this happens. When java memory mapped buffer memory replaces by os, we still have some free memory available.
JVM side configuration is tuned by our SREs and currently it is hard to further reduce. Why JVM size heap size will influence share memory which is in os page cache which is out of JVM management?
 
Old 03-16-2017, 04:01 PM   #4
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
File buffers are a form of shared memory but they are "low man on the totem pole" when memory pressure develops. If it is important to you that the recently-used files are served from cache for latency reasons, you need to control the total amount of working-set memory (active VM memory pages) that is called for. In a Java environment, fixing the heap-space limit (as previously described) is a standard fix, along with controlling the total number of processes that exist.

Obviously, you should also consider "throwing silicon at it." Can these servers accept more RAM? If so, buy it.
 
Old 03-16-2017, 04:37 PM   #5
newtonbear
LQ Newbie
 
Registered: Mar 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
What about if i call mlock to lock those memory mapped buffer?
Or i move those memory mapped file to a ram disk?
Buying new hardware is the last step to go.
 
Old 03-16-2017, 06:57 PM   #6
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
Hmm... the nodes thrashing for a reason, you likely already have a shortage of shmem, I wouldn't suggest you try to pin your pages, that seems like a good way to impact the efficacy of the underlying node, which would impact your java application more severely than any other strategy.
You could cgroup resources to your application or modify /proc/sys/vm/swappiness to say 8 or 10. You probably will have to tinker with it to find your "sweet spot", however I'm wondering if you couldn't just do a better garbage collecting your cache. That has the advantage of providing a solution without impacting the underlying node.
What stands out to me is you can dump it with minimal impact, that suggest it's not being efficiently collected. There is no upper limit to how much shmem it can use, so you're just pushing the issue back by pinning pages, you'll eventually run out of resources to pin.

Last edited by dijetlo; 03-16-2017 at 07:00 PM.
 
Old 03-16-2017, 07:11 PM   #7
newtonbear
LQ Newbie
 
Registered: Mar 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
What do u think about moving memory mapped file to a ram disk?
 
Old 03-16-2017, 07:12 PM   #8
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
EDIT: We're starting to get to the point with this where I'd need to know more about the virtualization infrastructure you're riding on before I could help you with that question, generally speaking though, when you have resource contention the only way to address it is by increasing the resource or lowering the consumption. You can work a little on the edges to get a little better efficiency however cache aggregation will eventually hunt you down.

Last edited by dijetlo; 03-16-2017 at 07:17 PM.
 
Old 03-21-2017, 08:47 AM   #9
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
I think that the only effective way to address your problem is to map out just how you intend for the applications to use the available memory, and to constrain the applications so that they never attempt to use more. Whether it's for the shared file buffer or for something else, RAM is always used for "something," and your goal is to arrange things so that the operating system never perceives a need to swap anything out. The only way to do that is to control the demand for memory resources. The operating system is reacting to what they are demanding.

Also, purchase just as much physical RAM as these boxes will hold. If you can add RAM to them, do so.
 
1 members found this post helpful.
Old 03-21-2017, 09:10 PM   #10
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
It might be a big redesign problem, and so you may not be receptive to it, but you might look at something like Redis for your caching layer instead of a native JVM solution. It's an in memory data structure store, it's got a minimal footprint, a really amazing performance profile (which would help you with your dynamic memory issues) and some really advanced algorithms for determining with a fairly high level of confidence the presence/absence of specific key values in the store (cardinality) which significantly improves the efficiency of "is that still in here ?" look ups (hyperloglog hashing is accurate to within .003% at standard hash depth, if I'm not mistaken).
It also has a time-to-live functionality in the atomic key-store structure that gives you fine grain control over the persistence of data within the cache which has the benefit of significantly improving garbage collection in a dynamic, persistent caching structure. Memcached is a similar product with less fine grain control however it's easier to implement.
Just a thought, hopes that helps

Last edited by dijetlo; 03-21-2017 at 10:00 PM.
 
1 members found this post helpful.
Old 03-22-2017, 10:39 PM   #11
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
I would hugely "second the motion" with regard to what dijetlo just said here.

Even though you might not readily want to embrace the notion that there might be "a [major] redesign problem" in your company's near-future, I would strongly encourage you to take these words to heart.

I have an extremely-thin book on my bookshelf ... The Elements of Programming Style ... which contained the following maxim:
Quote:
Don't 'diddle' code [to make it faster ...]: find a better algorithm.
Fortunately, in these days and times, the odds are quite excellent that someone else has already been there, and has developed a (sophisticated) solution.

Indeed, it seems to me that your true root-cause situation is that "your present software strategies, as you have presently implemented them," are just-beginning to exceed the capabilities of your hardware. But, your present strategies are also intolerant of any such excursion – if only because they have never happened before.

"Such things, once they begin to happen, are only likely to become more frequent." You might be able to "throw silicon at it" for a very short while, but not for long. Very soon, you're going to need to "stop 'diddling' this thing, and fix it."

... and, in doing so, you should specifically look for "prior art," such as Redis, instead of "your own devisings." (If necessary, force yourself, and/or your team, to think that way!)
Quote:
Actum Ne Agas: "Do Not Do A Thing Already Done."
 
1 members found this post helpful.
Old 03-22-2017, 11:55 PM   #12
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,099

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
Quote:
Originally Posted by newtonbear View Post
I am more curiously about how this happens. When java memory mapped buffer memory replaces by os, we still have some free memory available.
I missed the significance of this when I was first reading this thread. The memory manager has a bunch of sysctls to determine what is an acceptable memory pressure and at what point the low and high watermarks are set. They can be temperamental - small changes can have large (adverse) effects.
If you truly do have free memory, and no other consumers on the box, using a RAMdisk or similar may be an option. However it does remove that storage from the general pool so be aware of @dijetlo's warning re pinning above.
Also the file will have to be treated as a real disk file rather than a memory object.

(long time since I've done anything in java, so the above may all be irrelevant).
 
Old 03-25-2017, 10:41 AM   #13
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
Quote:
Originally Posted by syg00 View Post
If you truly do have free memory, and no other consumers on the box, using a RAMdisk or similar may be an option. However it does remove that storage from the general pool so be aware of @dijetlo's warning re pinning above.
Also the file will have to be treated as a real disk file rather than a memory object.
No, the memory occupied by the ram-disk can definitely be swapped out. In fact, it is a strong candidate for being swapped out since it is assumed that many of the entries are not being used.

You fundamentally have a design problem. Or, you need to "throw silicon at it."
 
Old 03-25-2017, 05:24 PM   #14
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,099

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
Perhaps you should read this.
There are very few legitimate reasons to use ramdisk these days, but they are occasionally useful.
 
Old 03-28-2017, 02:34 PM   #15
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
Quote:
Originally Posted by syg00 View Post
Perhaps you should read this.
There are very few legitimate reasons to use ramdisk these days, but they are occasionally useful.
The "initial RAM-disk" that is a part of most Linux distros these days is hardly a general case. In fact, it does not apply to operations at all.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
memory page swapping DKSL Programming 7 05-08-2012 08:31 AM
>> page swapping << devil_ Programming 1 04-10-2010 05:59 AM
Swapping vs Paging vs Virtual Memory SBN Linux - General 4 04-05-2008 09:06 AM
RH 7.3 not swapping memory...? severedhead Linux - Distributions 4 05-28-2002 05:16 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 04:33 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration