Thanks for explaining that so clearly! I think I know how it works now
I tried using that formula with the 1920x1440 modeline to get 85hz and got 336.6.
Naturally it didn't work, and was also garbled though not as heavily but still not usable.
And 65Hz looks a LOT better than 60Hz. It's pretty unbelievable how much difference 5Hz can make :P
I'll try and optimize it as far as I can now. Thanks again for your help.
Edit: I now have 2048x1536@70Hz and 1920x1440@82Hz running. Sorry everyone for the long thread!