One thing to keep in mind is that under the X protocol the client/server model seems to like it is reversed. Frequently the server runs on the local machine while the client runs on the remote machine. Just remember that the X server owns the resource (the display) and allows access from remote processes/machines.
The term XFree usually refers to the entire X package, including the server and the Xlib client and a few apps and drivers. X.org, the recent fork of XFree86 is going to split the whole thing up into several smaller packages.