Yeah, you're right about danger. The RFB protocol that VNC uses doesn't have encryption or secure authentication, which means that anyone might able to snoop on the connection and see exactly what you're doing on the remote machine, as well as impersonate you to connect to that machine in the future. Please do yourself a favor and use
SSH. For servers, it's generally better to skip GUIs and stick to command line tools anyway. If you must use GUI tools, use
X11 or
VNC tunneled through an SSH connection. Again, please don't use an unencrypted protocol like plain RFB (for VNC), telnet, or FTP to administer remote systems unless you have complete control of the network and implicitly trust everyone who ever might be able to connect to it.