Did you use
or something else? Anyway, that is how you can modify your XFree configuration safely in Debian.
It sounds like you chose video driver that can't do 1024x768... perhaps you chose "vga" from the list? You'll have to choose a right driver for your graphics card. Or perhaps you chose too low refresh rates (horizontal and vertical) for your monitor.