(Can I) use rpms for let's say Mandrake or Fedora with my RH Enterprise linux
short answer: no!
longer answer: maybe.
Generally if you are lucky, the binaries (and libraries) were built against the same files you have already installed. The idea behind RPMs is that you can drop them onto your system and it will find all files it depends on (those it is missing are also packaged in the same RPM).
So, if you install an RPM on a system it wasnt built for you can end up with unsatisfied dependencies. (You may be able to fix some problems by creating symlinks to newer versions of the libraries it needs, not all though.) What you have stumbled into is what developers sometimes call RPM hell (and they do that for a reason
In cases like that programs usually either work or they dont. So a malfunction of xmms is probably not because of RPMs from another distribution.
Hope this answers your question,