Do you have kernel-headers installed as well? And run
to see if you have correct versions. Also run
to see what kernel is currently active, so see if you have kernel-devel and kernel-headers for that particular kernel version.
I installed VirtualBox rpm's on several CentOS boxes and I believe that is all you need.