You definitely don't need the kernel source, but you really do need the kernel-devel package. That's what's installed on my AMD64/FC4 box, and I build the nvidia drivers without a problem after every kernel upgrade.
Verify that kernel-devel is installed by "rpm -qa | grep kernel-devel"
If it's not, download it and install it. If you use yum, then "yum install kernel-devel" will do the trick.
and install it using the command "rpm -Uvh kernel-devel-2.6.12-1.1447_FC4.x86_64.rpm"
I'm not at home right now, but I can check my config when I get home tonight and see if something I find might help you.