You might also see speed increases if, for example, the rpm was compiled for a Pentium, and you compile for a Pentium 4. I don't know the details, but my understanding is that the Pentium 4 has some instructions that can perform certain operations faster than the original Pentium. If you don't compile with the option to use those instructions, you'll just be running with the old Pentium instruction set. This is the difference between rpms labeled i386 and i686.
(Correct me if I'm wrong, anyone, or just provide more details.)