( I know this is an old post, I am responding just for people doing searches and to remove it from the 0 responses list)
Compiling FC It is different for different versions (FC1, FC2, FC3)
Read the particular release notes for your version
here
I don't think that you will get a noticeable speed boost by removing driver options from the kernel it may create a smaller kernel, but you have enough memory.
As for servers just disable the services.
If you want a distribution meant to play with the kernel, try
Gentoo . Fedora works best when configured as provided.