To see what modules are loaded:
Then you can (try to) remove one of those modules from the kernel with
is the name of the module you want to remove, that you saw in the 'lsmod' list. The 'lsmod' list does not, of course, include those 'drivers' that are built into the kernel (those that are not built as separate modules). If you want to remove a 'driver' that is built right into the kernel, you will need to recompile the whole kernel, leaving those things off that you don't like to have there.
To prevent a module from being loaded during boot process you can use, for example, module blacklisting; this means adding a line like "blacklist modulename" (where 'modulename' is the module you want to blacklist) into your module loading configuration file.