If you need a module, it is best to compile it into the kernel. See
http://www.linuxchix.org/content/cou...acking/lesson2. It will always be faster loading if the necessary modules are already a part of the kernel. For those things that are needed rarely, only run with specitic programs or tasks that are only occasionally used, one might consider that some savings would be realized, but most modules run for a purpose do not unload themselves and programmers often do not clean up. Then they are left loaded and take up unneeded space.
But considering the cost of memory, I think it is a false hope to try to save any actual cost by skimping on the kernel unless you have a need to conserve memory. You really don't end up saving much if any. So I suggest you just compile into the kernel any routines you use even just occasionally and you will be better off. Boot time for an extra megabyte is almost unmeasurable at the console.
The only caveat is that your kernel is larger and monolithic and might be more difficult to upgrade. I haven't experienced that and find the convenience of not having to call for module inserts makes most jobs easier. The monolithic nature makes module upgrades require a recompile of the entire kernel. So what? a few minutes more? Big deal.