Tracking the source of a killed job
 

Dear forum,

I am running calculations on a linux (redhat 8) cluster. We submit our jobs to an NQS type queuing system. I suspect that someone possibly w/ root permissions is killing my jobs. Is there any way I can track this possibly nefarious activity?

thanks,
gorets

Not without kernel patches. Killing is done through inter-process signaling, which is handled by the kernel. Process A tells the kernel "send a SIGHUP to 1234." The kernel then tells process 1234 "Hi, I have a SIGHUP for a process 1234."

Re: Tracking the source of a killed job
 

The queuing system almost certainly produces logs. You can grep the logfile for your job id. Alternatively, many queuing systems provide a job history command, which might provide some details.

Oops... I didn't think about the clustering software. :-P