You're correct in assuming X is the base. X simply sets up the GUI subsystems, whereas Gnome/KDE set up the "pretty" stuff, like taskbars and so on. As for your older system, it depends on both your expectations and desires. KDE and GNOME are desktops, typically packed to the gills with features. That means they are slower than others. Blackbox and other window managers are simply window managers, meaning they sacrifice a number of things for speed.
It's pretty much a try it and see deal. You can load all the desktops/window managers side by side, and play with them as you desire, till you find the best balance. On the other hand, "old" might be relevant as well. If your computer is better than a pentium II 233, it might provide as much speed as an XP box, even with one of those "bloated" desktops, if you use it more as an office-style computer. On the other hand, ir your idea of slow is a pentium IV, Linux is likely going to make Windows look pretty bad, once you factor in the number of applications available.
More info (not that it will make sense by reading, but links are always good):
Hope this helps a little.