I wonder, should the problem be xmms related, or system related ?
Have you tried other music readers ?
BMP (beep media player) is a fork of xmms for instance, they are very close, does he bug too ? KDE's Amarok offers the choice for your playing engine, you could try too. If you get the same problem with those, then xmms isn't "guilty"...