What distribution are you using? If your using Fedora, don't use the drivers directly from nVidia. If you're using Gentoo (which you're probably not), there's a guide on Gentoo's website. Also, the nVidia module version has to match the kernel version. You'll have to upgrade your nVidia module whenever you upgrade your kernel.
Anyway, kernel source files are typically in the /usr/src/linux path.
EDIT: I guess I should've looked at the forum you were posting in. Go
here for instructions for FC5 or
here for FC4.