Don't remove it just yet
. Install a new Window Manager as black(flux)box and see if that makes a difference. I think you can install either with apt-get
Also, another thing that works for me is, the system seems to be relatively more responsive when you boot into the command line and type "startx" ratter then having a cute "Welcome" screen as GDM, KDM, XDM, etc...
Try checking that too