Well, you did'nt post your computer specs.
I suspect you either have something running in the background using up resources or you have a small or inadequate processor. If you have amarok visualizations, that uses up allot of resources, so does amarok as it is a resource hungry app.
I seen this happen with Windows Media Player on a computer with a P4 2.6Gz processor when the monitor was plugged into a cheaper nvidia graphics card with no memory of it's own, when I plugged the monitor in a better nvidia card that had 128mb of it's own memory it would not happen. The music did'nt skip like yours but the graphics lagged severely when I would move windows around on the desktop, suggesting the processor is near maxed out, but when I minimized the Media player into the panel (not showing the visualizations), again, that would help. The visualizations use allot of resources.