Don't worry about your kernel being *too* big. The average kernel size is a little more than a megabyte. Almost always, you'll want things in the kernel (if possible) rather than to load modules -- it's almost always faster. Although, you should take out anything unnecessary:
--If you don't have FireWire, disable the firewire... if you aren't running any scsi devices, disable scsi. If you're not running a bus mouse, then don't bother including it in the kernel.
This will make your linux box boot up faster, and i believe also run faster, too.
Things to ALWAYS ALWAYS ALWAYS ALWAYS do when recompiling:
-- make sure that you're compiling for the correct processor (i.e. Pentium 1, 2, 3, Athlon, etc.)
-- make sure you allow support for the filesystems you're planning on using (ESPECIALLY ext3 and ISO9960 (or whatever the heck the CD one is)
there's more, but i'm not gonna list the whole doc for you. Though i'm sure you probably already have it, check out the
Linux Kernel HOWTO. It's got all the information you'll need for compiling the kernel and what to do afterwards.