Do this:
- From the host, use the route command to be sure that there is a route leading to the virtual machine's adapter. (You will see it being handled by a virtual-device that is obviously associated with VirtualBox.)
- Make sure that the sshd daemon is, in fact, running on the VM.
- Check the configuration file for sshd to be sure that it is listening to that adapter. (Sometimes, it's only listening to itself, e.g. 127.0.0.1.)
"Connection Refused" occurs when the port is not open: when there's no one on the other side listening to it, therefore "nothing to connect to."