You should not need to start X on the server. The Xserver runs on your local computer (VNC here, I think). The X programs are considered clients and do not need to be run on a machine which is running a server.
Add the following to your .profile of the remote system or type it when you log in.
set DISPLAY=184.108.40.206:0.0; export DISPLAY
Where 220.127.116.11 is your local ip address.
And make sure you permit the remote server to connect to your Xserver.
Where 18.104.22.168 would be the ip address of the remote server in this case. This is the X command to do this, I do not know about VNC.
FYI, an actual X server requires some input device or another. The X server terminology is often the reverse of what people expect. The XServer runs on the machine with the keyboard, mouse, and monitor. The XClients are everything else (from the login manager, window manager, and programs). These do not need to be on the same machine as the server -- although they often are on home machines -- and connect just fine over a network.
The network overhead for X is pretty heavy. If it's a local network, that's not bad. You're going to need a high speed connection if it's not local, if you want it to be comfortable to use.
I would recommend that you use command line tools whenever possible (which should be always) but if you insist on the graphical tools... just remember that you don't need to run XServer on the server, as odd as that sounds.