Glad to see I'm not alone
Have the same problem with my XBMC Slackware-current based machine and some cheap Logitech wireless keyboard/mouse combo. Some weird power management event disconnects it to the point that even unplug/plug sequence doesn't help. What helps is moving a mouse once in a while, which apparently stops the inactivity counter. I've never put any effort to the problem, but might try. The problem occured also on the older machine, which was based on XBMCBuntu distro, so we can safely rule out kernel or hardware issue. Therefore I agree with gengisdave's assumption - it's an x.org wrongdoing.