if your machine starts up the X on boot (and u don't have to run startx or xinit or Xsession ... and so on) you must reconfigure the rcinit files manually or with rcconf not to allow x to start on boot.
if you do need a desktop manager or just a window manager you can edit the .xsession file in your home dir and put in there "gome-session" if u want to run gnome or any window manager.
I just discovered enlightment

(wm) and I love it.
it has good integration with gnome, but I don't think that it uses very few mem resources.
run dselect(I kinda' like it) from console(if gnome-apt is running slow) and search window manager ( "/window manager") to look for a lightweight wm
after that u can reinsert the xdm daemon again if u like/want to run the X on boot.
hope I've been helpful