With Linux, for all packages/programs, you most often get three types of distributions:
1. Binary distribution
This means that the file(s) you download are binaries, i. e. programs (Windows programs usually come only in this type of distribution - i. e. Windows Binary) that you can just run and use.
2. Source distribution
This means the file(s) you download are source files (usually C/C++) that must be 'compiled' on your system to get them to work. They usually require 'libraries' that you are responsible to provide.
3. Development distribution.
This means the file(s) you download are for development - i. e. they are libraries that you can compile, that are then used by other libraries, that are used by source distributed programs that need them to compile.
This is what you want - a "development distribution" of Xwindows. Check the xfree86 website for the relevant X "development" libraries, or reinstall your distro and explicitly specify XWindows development files to be included in the installation.
On my Rh9 install, the X libraries had to be specifically included in a section called X Development, and took about 42 MB (I think!) to install. Without them I would not have been able to compile several programs that I am using now.