Process killed on linux unexpectedly

klnvsk · 06-23-2014, 09:10 AM

My process gets killed on linux server without manual intervention. I have verified the following scenarios.

•No manual intervention done to kill either by user or admin
•RAM and SWAP are not exhausted
* dmesg didn't provide any useful info
•'strace' gives me the message that "+++ killed by SIGKILL +++" - This makes me to assume that Kernel has killed the process.

The process occupies 16.5GB of virtual memory where the RAM is 16GB and the SWAP being 50GB.

Tasks: 393 total, 2 running, 387 sleeping, 4 stopped, 0 zombie

Cpu(s): 12.8%us, 0.5%sy, 0.0%ni, 86.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 16015M total, 8700M used, 7315M free, 202M buffers

Swap: 51199M total, 150M used, 51048M free, 5906M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

31466 xyz 20 0 16.4g 1.3g 6704 R 100 8.3 0:16.99 test

Could you please tell me the other reasons for Kernel to kill the process..

rtmistler · 06-23-2014, 02:57 PM

You could try something like this:

Code:

/*
 * Signal handler for kill signal
 */
void sigkill_sigaction(int signal __attribute__ ((unused)), siginfo_t *si, void *arg __attribute__ ((unused)))
{
    log("Caught sigkill at address %p, sent by pid: %d\n", si->si_addr, si->pid);
}

int main(int argc, char **argv)
{
    struct sigaction sa;

    /* Set up to catch sigkill */
    memset(&sa, 0, sizeof(sa));
    sigemptyset(&sa.sa_mask);
    sa.sa_sigaction = sigkill_sigaction;
    sa.sa_flags   = SA_SIGKILL;
    sigaction(SIGKILL, &sa, NULL);

    ...
}

Note that the si_siginfo structure has other info you can use (from sigaction(2) manual page):

Code:

       The siginfo_t argument to sa_sigaction is a struct with the following elements:

           siginfo_t {
               int      si_signo;    /* Signal number */
               int      si_errno;    /* An errno value */
               int      si_code;     /* Signal code */
               int      si_trapno;   /* Trap number that caused
                                        hardware-generated signal
                                        (unused on most architectures) */
               pid_t    si_pid;      /* Sending process ID */
               uid_t    si_uid;      /* Real user ID of sending process */
               int      si_status;   /* Exit value or signal */
               clock_t  si_utime;    /* User time consumed */
               clock_t  si_stime;    /* System time consumed */
               sigval_t si_value;    /* Signal value */
               int      si_int;      /* POSIX.1b signal */
               void    *si_ptr;      /* POSIX.1b signal */
               int      si_overrun;  /* Timer overrun count; POSIX.1b timers */
               int      si_timerid;  /* Timer ID; POSIX.1b timers */
               void    *si_addr;     /* Memory location which caused fault */
               long     si_band;     /* Band event (was int in
                                        glibc 2.3.2 and earlier) */
               int      si_fd;       /* File descriptor */
               short    si_addr_lsb; /* Least significant bit of address
                                        (since kernel 2.6.32) */
           }