XFree86-4.x.x polls the monitor directly and if a specified resolution is not supported by the hardware it defaults to the highest supported resolution in /etc/X11/XF86Config-4. If you use XFree86.3.x.x you need to specify your monitor specs, see the video timing
How-To for a complete explanation. I think you just had a faulty monitor.