You can build the module once for a kernel version and use it to load with different kernel configurations of the same version, if the module you are loading does not have any dependency with respect to the kernel code changes/configurations you do.
When you do "insmod <module_name>", while inserting the driver, this will probe for any dependency, version mismatch, module info etc.
If the different configurations of the same kernel version that you use have any dependency over the module you want to load, you may face version mismatch problems.
To check the dependency, you can can try "modinfo <module_name.ko>.
This will give you some useful info :