I can't think of any good reason to even run a X11 desktop on the server. To keep the server as lightweight as possibly, it probably doesn't even need to by running X on the console. You should be able to change the default runlevel to not start xdm at all (from 5 to 3 on most distros I think, but could be different for yours).
You could then just use your current mechanism (ssh -X & an X server on Windows) when needed.
If you need occasional access to X on the server, you can start it up by changing runlevels on the fly ("telinit 5" to start X; "telinit 3" to stop it again on my machine) - no need to reconfigure or reboot.
If you do need remote desktop for some reason, check out TeamViewer (http://www.teamviewer.com/