check out /var/log/XFree86.0.log for details on what went wrong...post at least the end of the file here if you can't figure anything out yourself
btw. as far as I know, there is no need to restart the machine after installing, say, nvidia drivers...only X.