Well I'm afraid XGL (which is still at development state if I'm right) eats up
a lot of resources; shortly said, you're not doing anything wrong, you just probably can't run games as smoothly as without XGL. You could use top and such programs to try and see how much resources it eats up, but my point of view is that it's eating a lot, and that since you don't need it for games (heh, it's not doing anything for your games -- just for your desktop
), don't use it while playing them. With a little effort you could configure things so that you run your games with regular Xorg and when not playing, use XGL. You seem to have a lot of memory, both system RAM and on video card, but I don't think it's enough for running XGL
and games.
I said it and I'll say it again: you're not winning anything having XGL running when playing games, so if you're wise, you don't buy more hardware but just don't use XGL when playing games.