I'm not sure, but i believe i've read somewhere that you don't have to recompile the whole kernel when you want to compile a module.
Just run make menuconfig (or whatever is your choice of configuring the kernel), select the module you want to compile. Save the kernel options, make dep and make modules && make modules_install. The kernel will not be compiled, only the modules will.
But then again.. I might be wrong
And so that I can learn something too:
IF I'm wrong, please tell me!