Personally, I use TigerVNC and have my clients set to autostart a VNC server that allows connection to the running display with
From my server, I can connect to a client from a terminal window
. (I actually use aliases to minimise typing.)

SSH connections between the server and clients have previously been established using passwordless keys. The VNC traffic is tunneled through SSH.

if you start up 2 or more VNC servers at the same time how do you switch between them? can you bind that switch to a 1 or 2 key action on the keyboard? if you have done this in some way, does everything in the non-viewed session keep on running with display change activity?

if they are remote sessions they will just work next to each other. If you start them on the same host their X servers should be connected to different consoles and in that case you can switch them by switching the console (Ctrl-Alt-Fx).
And yes, in this case they will work next to each other without any problem, just you can use only one of them (so mouse/keyboard/display is connected to the current/actual console). But if you use it on a local environemt you do not need vnc, just X servers.

i am looking at using some components of VNC to implement other capabilities.

i expect VNC clients all have some means for local user intervention, such as to exit the VNC client or to switch the input and output to a different VNC server that already connected, or to connect a new one. for example i could run a client on laptop6 and have connect to both server2 and server4. on the two servers the VNC port is listened by an ordinary VNC server program that lets users access X consoles. i just put these 2 clients in different windows and minimize the one i don't want to use at this moment and maximize the one i do want to use. this is what effectively happens with virtual workspaces. i typically run 10 or 20 of these per user across as many as 18 users for a potential total of 360 workspaces. but there are some occasional issues with all this so i am looking for other ways to do it.

my hope is that some VNC client will have a nice way to easily do this switch easily with limit based on available memory. one way to switch is bind particular keys to instantly do it within the client. another way is to detect mouse pointer movement to an edge and bring out a menu with a list of connections.

rather than modify a client to do this i am thinking of creating a VNC server that takes connections and interacts with the connected user to make client-like connections to other VNC servers. then that user can switch to whichever server is desired at the moment, leaving all the others up and running.

if you want to manage 360 workspaces/sessions you need to forget any kind of GUI, you have to use an automated solution, like puppet, ansible or something similar. I just can't imagine how you can handle that many windows on the monitor. (But that's my problem, not yours).
vnc client is just a window displaying something, like any other window (terminal, browser, whatever). If you want to have a special kind of switcher which can manage your vnc windows you need to implement it.

i am thinking about implementing it. VNC seems to be a simple enough protocol to do this with. the switcher will need a limited amount of GUI to implement a client facing login as well as a means to make connections. a client can logout (outgoing connections are dropped) or disconnected (outgoing connections are not dropped). a disconnected user can reconnect and continue using connections without the remote service seeing a network disconnect happening. this will be an added protocol over VNC (disconnect and switch view actions as well as non-interactive action to make a new connection. where to connect could be stored in switcher user logins, client, or user's brain, connected, then switched as the user prefers. existing VNC clients will just lack client stored list of connect places. switching action can simply be keystrokes the client passes to the switcher, such as Ctrl+Shift+KeyPad6 to switch to service number 6 (or whatever the user configures).

-Y is conceptually faster as it assumes a trusted connection and removes the overheads of X11 security extensions that -X otherwise uses.

VNC is faster than X forwarding and uses less bandwidth. I can watch youtubes under vnc that would be uncomfortable to watch under X forwarding.

VNC is nice as in how tmux is nice for being able to connect, do things, disconnect and having those tasks continuing in the background that you can later attach to again even from a different device. A gui version of tmux so-to-speak. I use one regularly at home from my wifi connected (older spec) 'couch' laptop to the ethernet wired (and powerful) desktop 'server'. The experience is that the laptop operates at the same speed as if I were sitting at the desktop system. My laptop boots using a kernel+busybox system, less than 20MB combined vmlinuz and initrd, so boots in a second or two. Within that it wifi net connects, has framebuffer vnc to connect to vnc servers, screen (similar to tmux), mc, calcurse (diary), full ssh for remote mounting/ssh server access, lynx textual browser, syncterm for old BBS's access ...etc. enough to use alone, but for chrome/libre vnc to a full gui desktop is appropriate. It also supports connecting my android phone as a hotspot, or for file transfers. I do also have Fatdog (Linux desktop system) and FreeBSD (X/gui desktop systems) as dual boot choices on the laptops HDD, I use the smaller system (usb stick) as a bootloader for those, can copy in the 512 byte MBR to the HDD according to whether I want to boot Linux or BSD.

Usable even if the internet was down, or was incredibly low bandwidth, could, at least conceptually use old dial up modem for a connection and still be usable other than not having chrome/youtube type availability due to lack of bandwidth. And secure as fundamentally its just outward ssh and vnc traffic flows.

I have four tty's set in the small boot, mostly the first is used for ssh tunnels etc, the next is set as a vnc/gui desktop (chrome/libre etc), the third has screen running which in itself provides additional terminal sessions. I use the 4th to ssh into yet other systems, or run syncterm (BBS's). IRC and chats etc. so its quicker to switch to rather than having those running within screen/tmux.

I use a variant of fbvnc, within which the default is ctrl-space to toggle suspend/resume screen updates. If I had two vnc sessions running then its just a matter of toggles turned on/off. I don't usually vnc into two different systems as I use that for just chrome/libreoffice type activities and whether that's served from a Linux or BSD or whatever server doesn't really matter, no/little need to have two such sessions running at the same time. I would imagine if that both were left toggled on/active then the screen would just present a merging of the two (a mess). Toggle one off and the screen would soon tidy up to showing just that vnc session screen content.

For confirmation, I've just tried it now, one vnc (fbvnc) session on tty ctrl-alt-F2, another on ctrl-alt-F4, with both left active (no ctrl-space fbvnc 'suspend') and on either of those screens I see a overlaid screen of the two gui desktop sessions, two mice (separated by a few inches in my test case - not aligned), right click the desktop and two menus pop up ...etc.). Ctlr-space suspend one and it restores to showing just the other active screen, ctrl-space suspend that, switch to the other (ctrl-alt-F4 for instance), and ctrl-space to activate that, and again just that clean desktop seen.

I believe more recent versions of fbvnc can include automatic suspend/activations according to whichever one you switch to, the source code I use to compile fbvnc however is old, not really had any need to change/edit it for years for my usage case.