Depends on you. You could probably get a little boost of speed with removing what you want, but its not really necessary.
The usual reasons for compiling your own kernel are to do with removing something that stops the kernel working with particular hardware on your machine, or to compile into the kernel drivers for diffrent hardware.
Oh and to say that you can of course
