Maybe you could create a script that runs at boot time with a high nice
value that would run in the background and automatically kill processes that exceed X amount of bandwidth or memory? Or at least start top at boot in the background and write values to a file once in a while???
EDIT: I just blundered across this
which may let you set some conditions for forced exit of processes.