Try renicing the Xserver process. It'll stop it hogging your cpu and let other tasks get a look in. I've found this can help with responsiveness when X is under heavy load.
You'll need to find the right process id and you'll have to run the command as root.
Code:
gazl@nix:~$ ps -e | grep X
2558 tty7 00:00:43 X
root@nix:~# renice +1 2558
2558: old priority 0, new priority 1
If it works for you then you can alter your Xserver to always run with this reduced priority.