LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-30-2012, 01:46 PM   #1
EpicOfChaos
LQ Newbie
 
Registered: Oct 2008
Location: Ogden, Utah
Distribution: Ubuntu Server 8.10
Posts: 18

Rep: Reputation: 0
Linux Process Reaped


Hello,
I have a process that will be running just fine for weeks at a time but after about two weeks of uptime the process while just be gone. I suspect it is exhausting all the memory on the machine, and since there is no swap setup it just kills the process. A long time ago I found a log that showed processes that get reaped because of memory consumption, however, I can not for the life of me find this log. Does anyone know where such a log would exist?

My Linux version : Linux 2.6.35.14-97.44.amzn1.i686 i686
(Amazon AMI)

I have checked /var/log
 
Old 10-01-2012, 01:51 AM   #2
nugat
Member
 
Registered: Sep 2012
Posts: 122

Rep: Reputation: 31
Typically, if a program has a memory leak, or is otherwise consuming all the system memory, it will just use it all up until the kernel seizes. If the program itself has memory management built in, then perhaps it is reaping itself, in which case perhaps it has its own logging facility. Or perhaps there is another generic process on the system that manages memory?

The syslog, typically /var/log/messages, is the first place I would look, but it sounds like you've already checked there.
 
Old 10-01-2012, 02:33 AM   #3
hydraMax
Member
 
Registered: Jul 2010
Location: Skynet
Distribution: Debian + Emacs
Posts: 467
Blog Entries: 60

Rep: Reputation: 51
Quote:
Originally Posted by nugat View Post
Typically, if a program has a memory leak, or is otherwise consuming all the system memory, it will just use it all up until the kernel seizes. If the program itself has memory management built in, then perhaps it is reaping itself, in which case perhaps it has its own logging facility. Or perhaps there is another generic process on the system that manages memory?
I do not think this is correct. During memory shortages, processes are killed by the Linux OOM (Out Of Memory) Killer as much as is necessary for the kernel to continue operating.

http://linux-mm.org/OOM_Killer
http://linux-mm.org/OOM

I'm not an expert on this, but I believe that it is all done by the kernel. There doesn't seem to be a separate log for it... I think OOM information just gets dumped to the regular kernel log which can be read by dmesg, but you should be able to filter out these messages to a separate file with syslogd if you wish.

According to the above references, you can mark a process so that it can not be killed by the OOM Killer.
 
Old 10-01-2012, 03:28 AM   #4
nugat
Member
 
Registered: Sep 2012
Posts: 122

Rep: Reputation: 31
Quote:
Originally Posted by hydraMax View Post
I do not think this is correct. During memory shortages, processes are killed by the Linux OOM (Out Of Memory) Killer as much as is necessary for the kernel to continue operating.

http://linux-mm.org/OOM_Killer
http://linux-mm.org/OOM

I'm not an expert on this, but I believe that it is all done by the kernel. There doesn't seem to be a separate log for it... I think OOM information just gets dumped to the regular kernel log which can be read by dmesg, but you should be able to filter out these messages to a separate file with syslogd if you wish.

According to the above references, you can mark a process so that it can not be killed by the OOM Killer.
OOM stuff should appear in dmesg|/var/log/messages, though, shouldn't it? something like

"Out of Memory: Killed process 18254 (ntop)."

that's why i suspected something else killing the process. the kernel is usually pretty clear when it is doing such things.
 
Old 10-01-2012, 10:09 AM   #5
EpicOfChaos
LQ Newbie
 
Registered: Oct 2008
Location: Ogden, Utah
Distribution: Ubuntu Server 8.10
Posts: 18

Original Poster
Rep: Reputation: 0
So it looks like you are right, the oom killer is what would log such and action. However, following the ideas on this link http://stackoverflow.com/questions/6...nux-oom-killer I am not showing my process was reaped. So most likely it is not consuming to much memory. Let me give you some more information, so the process that is running is a Glassfish 3.1.2.2 server instance, the logging that glassfish provides is not helping, one second it is serving requests like normal, than the next the entire process is gone. I am not sure where else to look for a solution.
 
Old 10-01-2012, 11:10 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,361
Blog Entries: 55

Rep: Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547Reputation: 3547
Maybe it would be better to run a small SAR (Atop, Dstat, Collectl, whatever else you fancy) and actually collect system statistics first? That, together with reviewing any bug tickets wrt Java and Glassfish and reviewing your Glassfish server settings might provide a more efficient approach because IMHO here looking for log entries is reactive, an after-the-fact op and that itself won't change or improve anything.
 
Old 10-01-2012, 11:44 PM   #7
nugat
Member
 
Registered: Sep 2012
Posts: 122

Rep: Reputation: 31
Quote:
Originally Posted by EpicOfChaos View Post
So it looks like you are right, the oom killer is what would log such and action. However, following the ideas on this link http://stackoverflow.com/questions/6...nux-oom-killer I am not showing my process was reaped. So most likely it is not consuming to much memory. Let me give you some more information, so the process that is running is a Glassfish 3.1.2.2 server instance, the logging that glassfish provides is not helping, one second it is serving requests like normal, than the next the entire process is gone. I am not sure where else to look for a solution.
Are you already starting it in verbose mode? e.g.:

Code:
asadmin start-domain --verbose
Documentation for that is here:

http://docs.oracle.com/javaee/6/tutorial/doc/bnadi.html

Apparently, there is also full-on debug mode available:

http://docs.oracle.com/cd/E19798-01/...afd/index.html
 
Old 10-07-2012, 07:33 AM   #8
markseger
Member
 
Registered: Jul 2003
Posts: 242

Rep: Reputation: 26
Haven't seen much activity here on this over the last few days. Any progress?
One thing about OOM I've discovered with collectl. Normally, collectl runs every monitoring interval at exactly the same time - no drift at all, well maybe an occasional msec, but it's very accurate and never misses a sample. Whenever OOM runs, at just about the highest priority as it can, a side effect is collectl stalls and missing sampling intervals. In fact on some systems I've seen collectl stall for over a couple of minutes! When this happens, you can often find a kernel daemon running at 100% in the process log either just before collectl stalls OR when I comes back, I forget which since I don't see it very often.
In other words, if you see long stalls in collectl logs, there's a good chance OOM was running and if there are no stalls it probably wasn't.
-mark
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
killng parent process without killing child process (Linux C programming) lettuce84@naver.com Linux - Newbie 3 07-24-2015 09:37 AM
Share glibc between Host process and a process within Linux Container alphy Linux - Virtualization and Cloud 0 02-16-2012 12:13 PM
Linux command to get Process name & Process executable path ArunChacko Fedora 2 08-26-2011 12:09 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:13 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration