According to the boys at #debian in the IRC, the driver is nvidia-glx and you need to compile the modules (nvidia-kernel-src), what i don't understand is why it doesn't work with vesa?
Assuming you are in Woody, with kernel 2.4.18, i would reconfigure X with:
and this time following the instructions in the following article TO THE LETTER:
The Very Verbose Debian 3.0 Installation Walkthrough
Sections 9 and 10
Installing XFree86 Parts I and II
try with the vesa driver and whatever you do, do not touch the XFree86config-4 file, Let Debian write it for you.