As root, do
let it finish then do
X -xf86config XF86Config.new
this should hopefully get you back in graphical mode so you can run the gui configurator again...
Under VMware I am able to get high resolutions, however you need to install the vmware-tools... so alternatively you can try this (you can do this from the command line):
First go in the VMware menu and do "install vmware tools...".
Then in the linux virtual machine do
mount /mnt/cdrom (depending on your distro, this might also be mount /cdrom)
tar zxf /mnt/cdrom/vmware-linux-tools.tar.gz
this should give no error messages; proceed with
Then follow the steps. I hope all filenames are still the same 'cos I have a slightly old VMware version.
If all is well it will have auto-adapted the graphical configuration and you'll be able to choose higher resolutions! I myself use 1280x1024 without any trouble!
Hope this works for you!