I had the similar problem. tried to switch from 800x600 to a better one as you did and got almost the same situation.

you need to boot to run-level 3, which can be done by editing grub menu. Login as root, try Xorg -configure. Xorg is something to configure X. Try man Xorg | grep configure (Does it really do so what the man page say?).

To get my system stable I tried Xorg -configure and also tried editing xorg.conf. With a several 'try' and editing, I ended up with 1024x768 plus thousand of colors. Since your X is already sick, you should not be surprised if the gui display stuff fails to work.

In my case, I boot to run-level 3, login and use startx. If I boot to graphical mood directly, X crashes at some point. I do not understand why. Probably because of driver.