I searched more and got several other Modelines:
Modeline "1280x768" 102.98 1280 1360 1496 1712 768 769 772 802
Modeline "1280x768" 94.98 1280 1352 1488 1696 768 769 772 800
Modeline "1280x768" 87.97 1280 1352 1488 1696 768 769 772 798
Modeline "1280x768" 84.75 1280 1328 1360 1440 768 771 778 796 +HSync -VSync
Modeline "1280x768" 80.14 1280 1344 1480 1680 768 769 772 795
I'm no expert on pixel clocks, refresh frequencies, and other Monitor related stuff, but I would think that an LCD monitor is more tolerant to these, as it internally transforms the input signal before use... I had the luck that the first ModeLine worked for me, so I can't help you very much.
But if I remember correctly there is a program which can generate the ModeLine for you...?
But as you say that it doesn't work, how does this manifest? Is the resolution just skipped, or is the picture distorted? Maybe the correct driver may be the problem?
Update:
I found an XFree86 Modeline generator. Since this is a basic function of X server, xorg should still be compatible with this...:
http://xtiming.sourceforge.net/cgi-bin/xtiming.pl