You have entered into the great topic of modularity vs non-modularity
.
While some argue for one, and others for the other, both have its advantages and disadvantages. If you compile, say, a driver support into the kernel, it will always be present and will be the very quick. But your kernel will get bigger for every driver you add, which finally will make it quite bloated. This will make your kernel slow, which was certainly not your original intention.
So you modularize the kernel, and load the modules whenever necessary
.