The file is
/etc/X11/xorg.conf if you're using Xorg. On XFree86 it's something like
/etc/X11/XF86Config or perhaps the same with
-4 added to the tail. Anyway, that's the file; and as you said, the resolutions are defined there, in the form of
"1024x768" "800x600" "640x480"
and so on. There are probably multiple lines, one line per one colour depth, but you should understand that. The way those lines are read is: the selected colour depth determines which one of the lines is used. In that line the first one from the left (above it's "1024x768") is the "default" one, i.e. the one that is first
tried to be used. If it cannot be used for some reason (card or driver not supporting it or something), then the next one is tried and so on until a suitable one is found or the list ends. This is what the man page says about it too, I think.
Two other important things are 1) vertical and horizontal refresh rates; these must match your screen, if defined -- wrong values may result in error or even damage something if you've got bad luck.. and 2) colour depth; use the colour depth you need, and one that is supported. All these are configured in the
xorg.conf file, in the sections for Monitor, Screen and even Device (video card).
After this you may have somekind of script that is read when X starts that tries to change the resolution; that's beyond X then. You should test the resolution with
which should start just X with xterm, with the desired resolution.