Oh yes, MySQL will hang your whole machine if someone wrote a faulty circular logic query that makes your mem usage climb to nowhere...
I'd highly recommend going through MySQL's documentation on administration. It's a neccessary read.
As for your clients who are hitting the DB server -- what client are they using and how are they creating their queries? are they using CLI? some 3rd party GUI?
MySQLCC has a nifty feature that escapes *some* memory intensive faulty queries. it also checks some syntax for you on the fly. I'd highly recommend it even to users. Just make sure their privs are set properly on the server.