Hello Sirs,

I have this server which had a OOM condition and killed few processes. Can someone please explain me how to find out which process got killed using the PID listed in the logs and how to see if the process was started back or not. OS is RHEL 6.x. Thanks in advance.

root@server102 [/root]
# grep -i 'kill' /var/log/messages
Jun 9 16:10:39 server102 kernel: ruby invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jun 9 16:10:39 server102 kernel: [<ffffffff81127892>] ? oom_kill_process+0x82/0x2a0
Jun 9 16:10:39 server102 kernel: Out of memory: Kill process 5343 (ruby) score 350 or sacrifice child
Jun 9 16:10:39 server102 kernel: Killed process 5343, UID 8162, (ruby) total-vm:2747456kB, anon-rss:1720308kB, file-rss:172kB
Jun 9 16:10:39 server102 kernel: vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 9 16:10:39 server102 kernel: [<ffffffff81127892>] ? oom_kill_process+0x82/0x2a0
Jun 9 16:10:39 server102 kernel: Out of memory: Kill process 5345 (ruby-timer-thr) score 350 or sacrifice child
Jun 9 16:10:39 server102 kernel: Killed process 5345, UID 8162, (ruby-timer-thr) total-vm:2747456kB, anon-rss:1720308kB, file-rss:172kB
root@server102 [/root]
# ps -ef | grep ruby*
root 9349 1 0 01:52 ? 00:00:02 /usr/bin/ruby /usr/bin/puppet agent
root 25056 23850 0 16:48 pts/0 00:00:00 grep ruby*
root@server102 [/root]

'Killed process ..' tells you what got killed.
You may possibly find other clues in other logs.

There were several many ruby based processes running (main lang puppet uses).

Processes do not automatically re-start unless special measures are taken.

You need to find the root cause of the OOM-killer being invoked.
Likely a memory leak or just too many (possibly large mem usage) processes being run.

You can use the various options to the top cmd to eg sort by memory usage; also a good idea to setup a monitoring tool