Is your kernel getting into a module loading race? Mine failed continually until I entered via-agp into /etc/modules so that it would load first.
Mine looks like this:
# /etc/modules: kernel modules to load at boot time.
# This file should contain the names of kernel modules that are
# to be loaded at boot time, one per line. Comments begin with
# a "#", and everything on the line after them are ignored.
via-agp #added to start agpgart first
If you are using Intel, you probably need a different module, but you get the idea.