Yes, don't use rpms do compile a kernel. They are bad enough for normal programs.
Get the sources from kernel.org and use the kernelhowto at tldp.org (or another howto on compiling a kernel). Make sure to get the newest 2.6 kernel, 2.6-test11, 2.6 is superior to 2.4. It's faster, and has a heck of a lot of more features.