What to do when kill -9 PID doesn't work
 

Due to a buggy external disk, programs accessing data thereon have a tendency not to respond to a kill -9. I thought -9 was putting it fairly strongly, but is there some other force mechanism? It's accessing the files read-only, so I hope it wouldn't corrupt the file system (I'm currently trying to copy the data off it).

Those processes are "not listening", thus the kill command cannot reach them. As a workaround you can try and umount the volume with -l option (see man umount).

I think there must be something wrong with the system other than a berserk process. It is my understanding the the -9 signal kills the process with no action required, or even permitted from the process. I don't think that the process needs to be listening in order to be killed. The OS handles it and forces termination, regardless of what the offending process is or is not doing.

Unfortunately, I'm still newbie enough not to know what to look for in your case. Have you tried restarting, or copying your files using a different distro?

This is all now academic hypothesis, but I believe because the mount process doesn't ever return, it assumes a zombie state. To kill/reap a zombie process like mount, I think I have to kill init (not desirable!). If someone in the know could better explain how reaping zombie processes work, I'd be grateful.

As for debugging the actual problem, it turns out it was the usb -> IDE controller that the disk was connected to. Hardware error.

Umm ... you can't kill init. The kernel won't let you. When a parent processes predeceases its children (or dies without waiting for completed children), init adopts them and cleans them up. This is part of the reason it's so vital for init to hang around (along with respawning ttys and the like).

However, that doesn't sound like the problem here. With dodgy hardware, processes trying to access it are often left in the D state (check top or ps). D is essentially uninterruptable sleep. This generally indicates that the process is executed in kernel mode, and because the kernel has allocated some resource for it (e.g. an I/O buffer), it cannot be killed because doing so could cause inconsistencies in the system. Therefore the process becomes unkillable, because its sudden disappearance could really trip up the kernel (e.g. the I/O buffer would be reallocated when the process died, so the hardware might write to an no-longer-valid spot in memory, causing general mayhem). Therefore the kernel "protects" it so that even kill -9 won't do it in. This is ordinarily supposed to be a very temporary state for a process, but when I/O hardware starts going dodgy, the operation the kernel is waiting on may take a long time or possibly never complete.

When this happens, you basically have two options:

(1) Hope that eventually some timer or interrupt will tell the kernel that this I/O operation just isn't happening. When that happens, the kernel will return an error to the process, move it out of D state, deliver the SIGKILL that you sent with kill -9, and the process will die. Depending on the nature of your flaky hardware, this may never happen.

(2) Shutdown the system, disconnect the fauly hardware, and power back on.

Uh, I know you can't kill init; my attempt at humour. (Huge kick in the comedy teeth there...)

Thanks for the excellent explanation in your second paragraph. I was using solution #2 until I got all the data off the disk.

I believe there is a signal that tells process to interrupt all its I/O attempts. Therefore the process becomes able to receive kill -9 from the system. I can't remember the exact signal to interrupt I/O, but it once (several years ago) helped me to kill k3b which hooked up while writing a disc.

OK, but what if I don't care to kill the process, and just want to release the terminal from it, so that to get back access to other shell jobs running in the background in the same session?

The problem is the hung process is in the foreground (active job in shell) and it doesn't respond to Control-C, Control-Z or whatever.

Note: I've got another terminal available, so I can issue any commands in the system.