I have seen this kind of post many times and have not really seen a good answer. Some of the things I remember to try are
Increasing the buffer size can help.
Renice xmms to a higher priority. But to renice it to anything under -1 to -19 requires root access to do. Check the man pages on nice.
Changing the output driver in xmms can help. If using arts try alsa.
Real Techincal Theroy is It has something to do with the IO scheduler under linux. This is a diffucult thing to discuss, but one can find more info on the web about this.
Some people have this problem and others don't. I have not seen any patterns whether it is cpu, ram, or versions of kernel, software, distros. I myself have no problems on any three machines I have.
If one or more items above work or find another answer please post your results here.
" Google the Linux way @ http://www.google.com/linux