I had this same problem with a Belkin OmniCube 2-Port KVM. Every time I switched to my new install of FC4, the mouse threw a wobbler. I soldiered on configuring the install, changed to KDM (from Fedora's default GDM) and I haven't had the problem since!
Of course, if I switch back to Windows and it's got Firefox as the foreground window, Firefox goes back one in the history, but that's a Windows problem