Just to be clear, are you talking about having these users have physical access to the host machine and then accessing the VMs from it, or having networked access to the VMs?
It would help to know a little more about what your are contemplating doing and what the surrounding computing environment is like. For example, if you are using bridged networking
for the VMs and there is a LAN, these users can have direct network access to the VMs without needing physical access to the host machine. You can then control their access to the VMs as you would any remote user's access to any computer. (If this is a feasible alternative, it would be my first choice, as it eliminates their needing access to the host machine.)