Debian by default sets all multi-user runlevels to invoke the GUI.
What I did was get sysv-rc-conf
and configure runlevel 3 not to start the gdm. It's in the repos.
I then needed to create a .xinitrc file in your /home/[username] directory for startx to start properly.
My .xinitrc contains one line:
It worked for me.