Actually, I think you have it backwards. Currently, X does run on your machine, and the problem is that in modern X apps the connection is very inefficient; so when they connect to your computer over the network the response is very slow. The solution actually is to not have X over the network and instead to have something like VNC or NX act as X on the remote machine and then VNC or NX transfers the information in an efficient way between the computers.
I don't know what you mean by you "can't support multiple users at once". Besides VNC, you can try NX (
FreeNX server for Linux), which is much faster than VNC (and up to 70 times faster than X), and has encryption and the standard user login mechanism.