The XFree86Config file is certainly the right place to add new video modes. The desktop settings in Gnome won't work if the X server doesn't know about corresponding modes.
The easiest way to do this is to reconfigure X-Windows using your distribution's setup tool, or (failing that) by running xf86config as root. Either of these will write out a new XFree86Config file, based on your hardware's reported settings.
If by “loopy” you mean that you can see the image on the screen but it won't sit still (shuffles back and forth or up and/or down, possibly with lines going across, and/or loss of colour), then you are running a mode whose refresh rate is outside the range your monitor can handle. Update the HorizSync and VertRefresh values in XFree86 with the values from your monitor's manual (HorizSync is often called the horizontal scan rate (MHz), and VertRefresh the refresh rate (Hz)). If in doubt, put 60 for the VertRefresh (most resolutions should work at 60Hz!), and then try increasing it (say 60-70 then 60-80) until you get something that looks better.
Hope that helps,
— Robert J. Lee