I looked up the envy, I assume you loaded envyng and not the envy. envyng is the one for hardy heron. Anyway.
When that freeze happens and you have to use the vesa driver, does your /var/log/Xorg.log file have any errors. If so, post the log file and people can look to see if anything sticks out.
I am imagining you are getting an error regarding cannot load nvidia module, but look and see. If that error is occuring, as I said post the log.
This is kind of wierd.