Sorry guys, just realized what really fixed it.
I did a fresh install and hadn't installed the Nvidia driver yet (I was recompiling the kernel). The real reason it wasn't locking up was because I was using the vesa driver. I just installed the Nvidia driver and the problem came back