Actually, I think you have it backwards. Currently, X does run on your machine, and the problem is that in modern X apps the connection is very inefficient; so when they connect to your computer over the network the response is very slow. The solution actually is to not have X over the network and instead to have something like VNC or NX act as X on the remote machine and then VNC or NX transfers the information in an efficient way between the computers.
I don't know what you mean by you "can't support multiple users at once". Besides VNC, you can try NX (FreeNX server
for Linux), which is much faster than VNC (and up to 70 times faster than X), and has encryption and the standard user login mechanism.