If harddisk space is not a problem, "installing Gnome but not using it", for example by always booting into the command-line multiuser runlevel, shouldn't eat up memory since Gnome is not running, especially if X is not running (and X is not running if you want to spare resources, right?).
If you need Gnome just for configuring stuff in GUI, and are afraid of consuming precious resources, you've picked up the wrong thing. Gnome is a big (read: huge) desktop environment, and whenever you're using it, you're using X under it and thus taking a lot of memory and some cpu power in use. Instead consider using a lighter desktop thing like XFCE or WindowMaker which enable you to run graphical tools but consume less memory (for example) than Gnome (or KDE). XFCE is even Gnome-looking.
You can use ssh with X11 forwarding to bring some windows/tools over the SSH to another X (or to Windows if you've bought an "X compatibility layer"), but to bring the whole Gnome Desktop you need another way; VNC might be one, but there are other choices too. I don't think bringing the whole desktop is sensible, it eats up a lot of resources, is slow or at least needs a very high-speed connection. I prefer plain SSH, or if you really need those graphical user interface windoes, use some very light X desktop on the server-end and then use SSH+X11 forwarding to bring on only the windows you need.