I can't say much about locally installed Mathematica, but I'd like to say this: I run Mathematica 5.2 from time to time over SSH and it's awfully slow that way (works, yes, but I don't like moving windows around nor scrolling anything like you said too). I thought it was because I run it over SSH, but I guess that's not the case then; also the connection speed I'm running it over is 10MB (yes, megabytes) so it shouldn't bee too slow
Sidenote: I have also tried running Maple and the other big number crusher over SSH, but they were soooo slow that I never actually got more done in a day then draw up the main screen, if using X11 forwarding like with Mathematica. Text mode works great, but isn't really helpful when dealing with images. Compared to them, Mathematica is fast as lightning with X11 forwarded over SSH, but still slow enough to make me go mad occasionally.
Oh, by the way I have used Mathematica 5.2 (actually just the same installation that I run over SSH) on a FC6 pc "on site", and it ran flawlessly fast there. Really odd.