Xen should have nothing to do with the display settings. Still you can boot into a non-Xen enabled kernel if you want to try running without it.
I'd suggest entering some very safe settings into system-config display, e.g. 800x600 resolution, low color depth, etc. just to see if that works. Then when you try to put back in your regular setting, look at the /etc/X11/xorg.conf file (which is the config file for the X server). In particular check the horizontal and vertical sync rates. If they are wrong for your monitor, bad things will happen.