Tracking the source of a killed job

gorets · 06-30-2005, 12:31 PM

Dear forum,

I am running calculations on a linux (redhat 8) cluster. We submit our jobs to an NQS type queuing system. I suspect that someone possibly w/ root permissions is killing my jobs. Is there any way I can track this possibly nefarious activity?

thanks,
gorets

Matir · 06-30-2005, 02:13 PM

Not without kernel patches. Killing is done through inter-process signaling, which is handled by the kernel. Process A tells the kernel "send a SIGHUP to 1234." The kernel then tells process 1234 "Hi, I have a SIGHUP for a process 1234."

jcliburn · 06-30-2005, 09:07 PM

Quote:

Originally posted by gorets
Dear forum,

I am running calculations on a linux (redhat 8) cluster. We submit our jobs to an NQS type queuing system. I suspect that someone possibly w/ root permissions is killing my jobs. Is there any way I can track this possibly nefarious activity?

thanks,
gorets

The queuing system almost certainly produces logs. You can grep the logfile for your job id. Alternatively, many queuing systems provide a job history command, which might provide some details.

Matir · 06-30-2005, 09:09 PM

Oops... I didn't think about the clustering software. :-P