I use Xfce on my old PII laptop and I do not use a graphical login at all. I boot the box to the console and then run startx if I want to start Xfce.
A graphical login really has no advantage; It is simply a prettier way to login. In fact, in some cases, I find it to be a disadvantage. For one, if you do a lot of kernel hacking, always running X is a waste of resources and can complicate certain tasks when you just want to run simple tests. In addition, I often just want to do some quick system admin tasks and I don't need X. By default I get six or so consoles, but if I boot to X I only get one. In short, if I don't need it I don't use it.
The main reason I see people needing GUI logins is if they have people that share their computer that would freak out if they did not have one.
I don't really understand why you don't like XDM. If you don't want to see the XDM login at all you can boot to the console. As I said before you can still shut down the computer from the Xfce GUI IF the permissions are setup properly (using groups or sudo).
If you don't plan on using the GNOME of KDE libraries I would not waste my time compiling and installing them.
If you have to pick between the GNOME and KDE login I would suggest KDE.
Also, if you would like to try a simple, very resource conscious distro that uses Xfce try Zenwalk linux (http://www.zenwalk.org/)
. The graphical login is already setup and looks nice (see the screen shots page). There is a liveCD, too.