Using the wait and defer arguments worked wonders. I also lowered the deferupdate on the Xvnc server down to 20 and that helped as well. x11vnc looks like it is the way to go.
Good, I'm glad cutting down the delays helped. In recent x11vnc's the default for both is 20 ms, but it used to be 30. I suggest 5-10 for local viewing.
Not clear how Xvnc's deferupdate would come into play here (since it is not doing the VNC), but maybe it makes some difference in the timing.
I checked and it does not look like there is a DAMAGE extension for the X server. I just looked at the command line options and did not see the damage option. If the damage extension is in a configuration file or something than I am not sure where to find it.
You would have to run "xdpyinfo" pointed at the display and you'd see it listed:
# xdpyinfo -display :20
name of display: :20.0
version number: 11.0
vendor string: The X.Org Foundation
number of extensions: 24
If your Xvnc doesn't support DAMAGE (however I think some versions do), an alternate is to use Xvfb. Recent versions of x11vnc automate much of this for you. E.g.
x11vnc -create -env FD_GEOM=2560x1024x24 -env FD_SESS=kde -clip 1280x1024+0+0
for the left x11vnc, and after
connecting to that one with a viewer (connecting creates the Xvfb desktop)
x11vnc -find -clip 1280x1024+1280+0
for the right one. This assumes you have no other X sessions on the machine (otherwise they will use it instead).