Debian by default sets all multi-user runlevels to invoke the GUI.
What I did was get
sysv-rc-conf and configure runlevel 3 not to start the gdm. It's in the repos.
I then needed to create a .xinitrc file in your /home/[username] directory for startx to start properly.
My .xinitrc contains one line:
It worked for me.