You will get better performance with a monolithic kernel than you would when using modules. If nothing else, you will be saving the disk I/O required to load the modules into the kernel.
The downside is that the kernel will become larger the more things you add in. This can be a problem if the kernel becomes so large it no longer fits in the MBR (such as if you were booting with LILO). Though you would need to include quite a few modules before that happened.
A safe compromise is to build in all of the modules you absolutely need, and then leave loadable module support enabled (along with the compiled source for your kernel). Then, down the line, if you want to play around with some particular module or function (say virtual tunnels or a USB device) you can quickly compile and install the module for it without having to replace the whole kernel (or even reboot).
Just having modules on the machine and leaving module support enabled is not going to cause any loss in performance, so long as your required support is built in. You would only be using modules when you actually required them, rather than full time.