LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-11-2014, 01:44 PM   #1
MirceaKitsune
Member
 
Registered: May 2009
Distribution: Manjaro
Posts: 156

Rep: Reputation: 1
Prevent memory leaks from crashing the system


The problem has reached a point where I'm no longer willing to let it slip, so I wish to find a solution to this ASAP. What happens precisely is: Occasionally, programs have memory leaks. Usually they tend to be harmless, as they are slow and / or don't fill quite a lot of RAM. In a few cases however, a process fills the memory with Gigabytes of RAM in a matter of seconds, until every bit of it is taken up. When this happens, the system becomes unusable (nothing can be clicked, mouse pointer freezes, etc) and the computer needs a hard restart. This is both a major problem as well as a security risk, because such a thing can cause loss of data or put the system in danger (consider it were to happen in the middle of a Kernel update).

I was hoping that at this day, the Linux Kernel has a minimal protection against this sort of thing, and enforces a pocket within the available RAM which user applications cannot simply take up... in order to save the system if memory is filled abusively. But since it doesn't appear to, I need to add one manually. Problem is I don't know how exactly, and I'm hoping someone can clarify me better.

What is the best way to limit memory usage for normal processes, in order to disallow memory leaks from bringing the system down by blocking all the RAM? I'm thinking something that restricts non-root processes to only part of the total memory. For example, I have 9GB of RAM. If it were to solve those crashes, I am okay with allowing 1GB to be used only by root and system processes, whereas normal programs I run may only have access to the other 8 GB.

The right path seems to be the ulimit command and the /etc/security/limits.conf file. But ulimit seems to have a lot of parameters and memory types it addresses (which aren't clarified either) and I'm not sure exactly what to set it to for this scenario. I basically seek the ulimit settings that require me to give up as little RAM as possible, in exchange for guaranteeing a space that memory leaks cannot touch to keep the system safe. Also, I'd prefer using percentages rather than fixed values, so I don't have to re-configure everything if I gain or lose RAM... like for example use "90%" instead of "8000 MB".

One clarification: I believe that in the past, I've heard people say that if a process of low priority has a limitless memory leak, it shouldn't actually take down the system because the Kernel knows to handle it, so maybe something else is happening. I've had the problem numerous times and can confirm this is false! If any badly written program fills up all the memory in a few seconds (which I get to see in KSysGuard before the system dies) it will render the system unusable and the user has to unplug the computer and start it up again. Also, I do have a SWAP partition... and large one at that (8GB). Even so, such leaks do bring down the system.

Last edited by MirceaKitsune; 07-11-2014 at 01:46 PM.
 
Old 07-12-2014, 06:46 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by MirceaKitsune View Post
(..) Occasionally, programs have memory (..) In a few cases however, a process fills the memory with Gigabytes of RAM in a matter of seconds, until every bit of it is taken up. When this happens, the system becomes unusable (nothing can be clicked, mouse pointer freezes, etc) and the computer needs a hard restart. (..) such a thing can cause loss of data or put the system in danger (consider it were to happen in the middle of a Kernel update).
This thread seems to be a continuation of this previous thread of yours which you marked "solved". Since it apparently isn't, I suggest you confine your posts to that thread. (I'll arrange this one to be merged there.)

Note "filling RAM" is an observation that only works if you don't know how the Linux (kernel) Virtual Memory Manager (VMM) works. If you don't please read up on it in kernel/Documentation/.

If this thread is distinctly different from the other one then I suggest you do not combat symptoms but address the cause. Said thread, nor this one, show any system activity reporting (SAR), process lists / specifics. Nor is it clear if you run any software that is prone to memory leaks or if you actually tried tracing those memory leaks. Additionally this thread doesn't follow up in terms of what changed since you marked the other thread solved with respect to switching back from the OSS Radeon to the proprietary ATI fglrx driver, changes to vm.swappiness or other VMM-related sysctls or enabling BIOS "Extreme Memory Profile" or any other (over-clocking) settings.

Last edited by unSpawn; 07-12-2014 at 06:47 AM. Reason: //typo
 
Old 07-12-2014, 08:09 AM   #3
MirceaKitsune
Member
 
Registered: May 2009
Distribution: Manjaro
Posts: 156

Original Poster
Rep: Reputation: 1
Oh... this is different from that other thread, which was a system issue with memory management in general. This is about how the Kernel behaves when a process fills all the available memory at once. So don't merge them please, I think they're completely unrelated problems.

Otherwise, it's hard to gather more info after such a problem takes place. Since once the leak is triggered, I only have a few seconds to move the cursor around, before slowly everything slows down and eventually gets stuck. I spoke about this in other places and people have clarified me on the matter some more for now.
 
Old 07-12-2014, 08:37 AM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
First, you need a definition of "killing the system". Allocating memory doesn't have diddly to do with killing the system. What happens is that the system starts thrashing...

I would suggest disabling oversubscription... and tuning with the overcommit_ratio value. There is also the swappiness and stat_interval that can be tuned.

http://www.win.tue.nl/~aeb/linux/lk/lk-9.html

A setting of 2 will prevent the process from "killing" the system. NOTE: 80G of swap will still allow the system to thrash. You will need a more appropriate size. Oversubscription is a guaranteed way to deadlock when it is done "right".

BTW, it isn't a "low priority has a limitless memory leak" that causes problems. It is a process that allocates HUGE amounts of memory and then uses it up "slowly" so as to avoid the OOM analysis.

The best solution is to identify the program, then fix it.
 
Old 07-12-2014, 11:44 AM   #5
MirceaKitsune
Member
 
Registered: May 2009
Distribution: Manjaro
Posts: 156

Original Poster
Rep: Reputation: 1
Yeah that's what I meant by killing the system; When the RAM gets filled, everything slowly slows down over the course of a few seconds, until I can no longer click anything and even the mouse pointed stops working most of the time. Sooner or later it becomes completely unresponsive and the image permanently frozen.

The settings you mentioned seem like exactly what I was looking for! I shall take a better look at them and possibly enable them later. Thank you for the useful info.
 
Old 07-12-2014, 12:25 PM   #6
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Ulimits can help - depending on the resource being consumed.

The defaults I see are:
Code:
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63715
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
The ones specifically a problem are the virtual memory and the data segment size. The normal control on virtual memory should cause the program to abort as soon as it exceeds this limit.

One reason ulimits are difficult to manage is that they only apply to a single process. Once it forks/threads it gets a duplicate of the limits. That is where the global limits on oversubscription come in.

Last edited by jpollard; 07-12-2014 at 12:27 PM.
 
Old 07-12-2014, 03:03 PM   #7
MirceaKitsune
Member
 
Registered: May 2009
Distribution: Manjaro
Posts: 156

Original Poster
Rep: Reputation: 1
Alright... until proven otherwise by a new leak, this issue should now be solved. After googling around for a bit based on the suggestion posted above, I added the following settings to sysctl.conf and applied them with the sysctl command for this session:

Code:
vm.overcommit_memory = 2
vm.overcommit_ratio = 90
I have 9GB of physical RAM (impair because triple-channel) and 8GB of SWAP. This means that 8.1 GB (90% of 9GB) + 8.0 GB = 16.1 GB is the highest amount of memory the system is allowed to ever use, leaving 1GB out for safety. That is more than enough for me, and exactly what I was looking for. Hopefully those settings are right... I'll see how it goes.
 
Old 07-12-2014, 06:15 PM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by MirceaKitsune View Post
Oh... this is different from that other thread, (..)
OK.


Quote:
Originally Posted by MirceaKitsune View Post
This is about how the Kernel behaves when a process fills all the available memory at once.
Part fuelled by a repeated and distinct lack of tangible information I still maintain you're combating symptoms instead of addressing the cause.
 
Old 08-06-2014, 06:09 PM   #9
MirceaKitsune
Member
 
Registered: May 2009
Distribution: Manjaro
Posts: 156

Original Poster
Rep: Reputation: 1
Ok... apparently vm.overcommit_memory isn't all that safe to use. After approximately 2 days of uptime, the KDE screen locker eventually breaks... with an error saying "unable to connect to authentication system" or something like that, whenever I type my password and press enter to unlock. And yes, I analyzed it several times... this only happens with vm.overcommit_memory = 2 (didn't try 1). I'm probably not going to bother bug reporting this... since it's likely one of those things that are hard to test and only happens to me and a few other people out there. I wish Linux (especially KDE) wasn't so buggy all the time though...
 
Old 08-07-2014, 05:20 AM   #10
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
The system didn't crash. It aborted a process it deemed was suitable.

You should look at the logs and see WHAT was aborted, and why.

All the overcommit does is prevent the system from a hang. It does not prevent memory leaks.

There are even things that can be done to specify WHICH processes are not to be subject to being aborted - but you also have to accept that the process with the leak may be one you don't want aborted...
 
Old 08-07-2014, 08:49 AM   #11
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
You should look at some of the parameters documented here:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt

You should set a larger value on admin_reserve_kbytes in order to have enough memory during the failure to log in on an admin account and get a good look at what is going wrong (and maybe fix things by killing the problem process).

I didn't understand min_free_kbytes well enough myself to advise you in detail, but it looks like a parameter you need to increase.

If you have a memory leak and not enough swap space to cover the leak, then something needs to fail. At best the parameters give you some influence over what fails. Messing with the over-commit settings is (as you have seen) an indirect and dangerous control relative to the problem you are trying to manage.

Last edited by johnsfine; 08-07-2014 at 08:50 AM.
 
  


Reply

Tags
block, crash, leak, memory, ram


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Xorg using extremely high amounts of memory (memory leaks?) HikoHaieto Linux - Software 2 04-07-2013 02:04 AM
[SOLVED] Are memory leaks possible? kb2tfa Linux - Server 5 09-29-2012 10:38 AM
Memory leaks.. *** glibc detected *** ./SuffixTree: malloc(): memory corruption: 0x00 evansash Programming 12 03-21-2011 01:17 PM
US urges action to prevent insider leaks Jeebizz Linux - News 0 01-05-2011 09:29 AM
Linux, an Operating System with Memory Leaks? czamora Linux - General 3 09-12-2006 02:34 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:51 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration