After many many googles... I have come to the conclusion that compiling your own kernel isnt that hard, but no one explains exactly whats going on. (well, sorta)
What I want to do:
1. Compile the kernel with the correct modules only needed for my hardware.
2. Compile a new kernel with just the basics to hit a prompt. Absolutely no "extra fat."
3. Install the packages I need to run things like x, word processing, email etc. for a simple workstation. (Because thats all I need it to do)
As quoted from Capt_Caveman
It's usually a good practice to either do a minimum install or to remove unneeded applications (ie you shouldn't need an mp3 player or GIMP on a router) as they provide possible vectors for compromise (can't exploit an application that doesn't exist).
It seems on most distros and installations, even the very minimalistic of installs gives you a lot of "extra" that you really do not need or use. And thats exactly what I want to eliminate.