Linux - KernelThis forum is for all discussion relating to the Linux kernel.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Our distributed server runs for sometimes.
We are setting /proc/sys/vm/swappiness as 0.
Our service is based on JVM with memory mapped file with java MappedByteBuffer through RandomAccessFile which in top command shown as shared memory.
Recently we find that our service some of shared memory got flushing out from physical memory which causes some latency issue.
Just wonder is there any condition that memory mapped file still can be replaced by os?
Does anybody know the reason for this?
And if this issue happens, if we just restart our service, this issue will come up in next several hours. The only way we find to resolve it is to drop the file cache by echo 1 > /proc/sys/vm/drop_caches.
The condition you're describing is a form of "thrashing", the processor is consuming more cycles moving data in and out of shmem than it is processing your workload. The most common fix is to limit your java heap at instantiation .
Quote:
The only way we find to resolve it is to drop the file cache by echo 1 > /proc/sys/vm/drop_caches.
If you can dump your caches into /dev/null without suffering process failure of one kind or another, you should definitely limit the heap at instantiation.
java -Xms<initial heap size> -Xmx<maximum heap size> javaApp in a bash whack is normally how it's done.
Hi dijetlo
Thanks for kindly replying.
I understands it is a kind of thrashing.
I am more curiously about how this happens. When java memory mapped buffer memory replaces by os, we still have some free memory available.
JVM side configuration is tuned by our SREs and currently it is hard to further reduce. Why JVM size heap size will influence share memory which is in os page cache which is out of JVM management?
File buffers are a form of shared memory but they are "low man on the totem pole" when memory pressure develops. If it is important to you that the recently-used files are served from cache for latency reasons, you need to control the total amount of working-set memory (active VM memory pages) that is called for. In a Java environment, fixing the heap-space limit (as previously described) is a standard fix, along with controlling the total number of processes that exist.
Obviously, you should also consider "throwing silicon at it." Can these servers accept more RAM? If so, buy it.
What about if i call mlock to lock those memory mapped buffer?
Or i move those memory mapped file to a ram disk?
Buying new hardware is the last step to go.
Hmm... the nodes thrashing for a reason, you likely already have a shortage of shmem, I wouldn't suggest you try to pin your pages, that seems like a good way to impact the efficacy of the underlying node, which would impact your java application more severely than any other strategy.
You could cgroup resources to your application or modify /proc/sys/vm/swappiness to say 8 or 10. You probably will have to tinker with it to find your "sweet spot", however I'm wondering if you couldn't just do a better garbage collecting your cache. That has the advantage of providing a solution without impacting the underlying node.
What stands out to me is you can dump it with minimal impact, that suggest it's not being efficiently collected. There is no upper limit to how much shmem it can use, so you're just pushing the issue back by pinning pages, you'll eventually run out of resources to pin.
EDIT: We're starting to get to the point with this where I'd need to know more about the virtualization infrastructure you're riding on before I could help you with that question, generally speaking though, when you have resource contention the only way to address it is by increasing the resource or lowering the consumption. You can work a little on the edges to get a little better efficiency however cache aggregation will eventually hunt you down.
I think that the only effective way to address your problem is to map out just how you intend for the applications to use the available memory, and to constrain the applications so that they never attempt to use more. Whether it's for the shared file buffer or for something else, RAM is always used for "something," and your goal is to arrange things so that the operating system never perceives a need to swap anything out. The only way to do that is to control the demand for memory resources. The operating system is reacting to what they are demanding.
Also, purchase just as much physical RAM as these boxes will hold. If you can add RAM to them, do so.
It might be a big redesign problem, and so you may not be receptive to it, but you might look at something like Redis for your caching layer instead of a native JVM solution. It's an in memory data structure store, it's got a minimal footprint, a really amazing performance profile (which would help you with your dynamic memory issues) and some really advanced algorithms for determining with a fairly high level of confidence the presence/absence of specific key values in the store (cardinality) which significantly improves the efficiency of "is that still in here ?" look ups (hyperloglog hashing is accurate to within .003% at standard hash depth, if I'm not mistaken).
It also has a time-to-live functionality in the atomic key-store structure that gives you fine grain control over the persistence of data within the cache which has the benefit of significantly improving garbage collection in a dynamic, persistent caching structure. Memcached is a similar product with less fine grain control however it's easier to implement.
Just a thought, hopes that helps
I would hugely "second the motion" with regard to what dijetlo just said here.
Even though you might not readily want to embrace the notion that there might be "a [major] redesign problem" in your company's near-future, I would strongly encourage you to take these words to heart.
I have an extremely-thin book on my bookshelf ... The Elements of Programming Style ... which contained the following maxim:
Quote:
Don't 'diddle' code [to make it faster ...]:find a better algorithm.
Fortunately, in these days and times, the odds are quite excellent that someone else has already been there, and has developed a (sophisticated) solution.
Indeed, it seems to me that your true root-cause situation is that "your present software strategies, as you have presently implemented them," are just-beginning to exceed the capabilities of your hardware. But, your present strategies are also intolerant of any such excursion – if only because they have never happened before.
"Such things, once they begin to happen, are only likely to become more frequent." You might be able to "throw silicon at it" for a very short while, but not for long. Very soon, you're going to need to "stop 'diddling' this thing, and fix it."
... and, in doing so, you should specifically look for "prior art," such as Redis, instead of "your own devisings." (If necessary, force yourself, and/or your team, to think that way!)
I am more curiously about how this happens. When java memory mapped buffer memory replaces by os, we still have some free memory available.
I missed the significance of this when I was first reading this thread. The memory manager has a bunch of sysctls to determine what is an acceptable memory pressure and at what point the low and high watermarks are set. They can be temperamental - small changes can have large (adverse) effects.
If you truly do have free memory, and no other consumers on the box, using a RAMdisk or similar may be an option. However it does remove that storage from the general pool so be aware of @dijetlo's warning re pinning above.
Also the file will have to be treated as a real disk file rather than a memory object.
(long time since I've done anything in java, so the above may all be irrelevant).
If you truly do have free memory, and no other consumers on the box, using a RAMdisk or similar may be an option. However it does remove that storage from the general pool so be aware of @dijetlo's warning re pinning above.
Also the file will have to be treated as a real disk file rather than a memory object.
No, the memory occupied by the ram-disk can definitely be swapped out. In fact, it is a strong candidate for being swapped out since it is assumed that many of the entries are not being used.
You fundamentally have a design problem. Or, you need to "throw silicon at it."
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.