My issue is that if one or both of the TV's get turned off I lose the placement of the display.
Sounds like a problem I ran into with a windows remote access client... the one bundled in the Symantec trainwreck, I forget the name of it.
It reset monitors, especially dual monitors, because it reset video display settings in order to give me access. Probably doing the same thing to your Xserver.
The fix would be to set up a remote access Xserver config as an alternate and secure shell in and swap it with the existing Xserver set up. That way, when you exit, you can swap them back and the good configuration is unmolested by your remote shenanigans,