Do you have authorization to run on your x server (when you're on the console)?
To find out, go to the non-graphical console, and type
If you get error messages, you probably don't have authorization. If you check your screen, it should be running an xterm now.
To get authorization depends on your distro. The most basic way is use xauth and MIT-cookies to do something like
xauth add 0:0 . `mcookie`