LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-09-2015, 05:07 PM   #1
rookee
LQ Newbie
 
Registered: Jun 2014
Posts: 28

Rep: Reputation: Disabled
Finding out information about a process that got killed.


Hello Sirs,

I have this server which had a OOM condition and killed few processes. Can someone please explain me how to find out which process got killed using the PID listed in the logs and how to see if the process was started back or not. OS is RHEL 6.x. Thanks in advance.

root@server102 [/root]
# grep -i 'kill' /var/log/messages
Jun 9 16:10:39 server102 kernel: ruby invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jun 9 16:10:39 server102 kernel: [<ffffffff81127892>] ? oom_kill_process+0x82/0x2a0
Jun 9 16:10:39 server102 kernel: Out of memory: Kill process 5343 (ruby) score 350 or sacrifice child
Jun 9 16:10:39 server102 kernel: Killed process 5343, UID 8162, (ruby) total-vm:2747456kB, anon-rss:1720308kB, file-rss:172kB
Jun 9 16:10:39 server102 kernel: vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 9 16:10:39 server102 kernel: [<ffffffff81127892>] ? oom_kill_process+0x82/0x2a0
Jun 9 16:10:39 server102 kernel: Out of memory: Kill process 5345 (ruby-timer-thr) score 350 or sacrifice child
Jun 9 16:10:39 server102 kernel: Killed process 5345, UID 8162, (ruby-timer-thr) total-vm:2747456kB, anon-rss:1720308kB, file-rss:172kB
root@server102 [/root]
# ps -ef | grep ruby*
root 9349 1 0 01:52 ? 00:00:02 /usr/bin/ruby /usr/bin/puppet agent
root 25056 23850 0 16:48 pts/0 00:00:00 grep ruby*
root@server102 [/root]
 
Old 06-10-2015, 05:13 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,300

Rep: Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358Reputation: 2358
'Killed process ..' tells you what got killed.
You may possibly find other clues in other logs.

There were several many ruby based processes running (main lang puppet uses).

Processes do not automatically re-start unless special measures are taken.

You need to find the root cause of the OOM-killer being invoked.
Likely a memory leak or just too many (possibly large mem usage) processes being run.

You can use the various options to the top cmd to eg sort by memory usage; also a good idea to setup a monitoring tool
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
PID information after killed deepak_message Linux - Server 1 09-21-2014 03:30 PM
Does child process get killed ,when we kill parent process.? gaurav_s Linux - Newbie 1 08-01-2014 12:50 AM
stopped processes - finding out who/what killed them? simulator Linux - Newbie 11 03-30-2012 01:09 AM
Finding Killed Process Harlin Linux - Security 3 11-01-2007 03:04 PM
find history of a job killed by "kernel: Out of Memory: Killed process" poulacou Linux - Server 3 09-20-2007 04:24 PM


All times are GMT -5. The time now is 07:11 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration