This happens at random times? Is it always when you're browsing the web? If it has only happened since you upgraded your kernel, that could be the problem. Your best bet is probably to compile the kernel yourselfm since the precompiled one might have stuff in that you don't want, which might be causeing it. Can't think of anything else particularly though.
Also, when you say it freezes completely, have you tried pressing Ctrl-Alt-F2 (or in fact any of F1-6)? This should take you to a console, where you can log in as root, type "ps -e" to list all processes, find out where X is running, and kill it (with "kill pid", where pid is the process ID). Then you can restart it with "startx".