In Redhat's case you are loading precompiled binaries. The source code available as src.rpm is there, too; but you are loading a binary.
This is generally a non-issue. The speed difference between arch_386 and i686 is generally unnoticable in the scheme of things. The slowest resource in the system is you.
The advantage of compiling from source is that you will 1) know what's there; 2) know where it is and 3) usually know why it is there. I think that a good LFS build or a gentoo build (though gentoo is non-standard(!) about some scripts and such) is well worth the effort. YMMV.