Well the prob might be that you havr the wrong settings on your vga side of things. I suggest that you select a different monitor type. If u have selected the correct monitor type then check to see if you have selected the right vga card. If both are correct then try using the generic setting for both. Last but not least try going to RH web site and see if your hardware is supported buy them here is the link http://hardware.redhat.com/hcl/.
One more thing if everthing was properly selected from the begining like vga card and monitor and u changed to 1024x768 then try using less colour depth and see if it will accept it.