It's time for you to do a risk assessment. You need to decide how critical your data is and what access methods are required/appropriate. Security and convenience are usually on different ends of the scale. If this is just a personal site, you might be willing to accept the additional risk. However, if this is a corporate server, then I would probably pass. If you do decide to use VNC, you might consider using
TightVNC. It supports SSH tunnelling for Unix platforms.
VNC on a server
