Just start the X-server with the correct arguments. (Look at /etc/X11/xinit.rc - or a similar location/name - for examples.) Or look at how the
startxfce4 program does it after it's invoked by the login application. (Usually in
/usr/bin/startxfce4, but it is a shell script you can
cat.)
Note that you still need a display manager (usually
gdm or
kdm) started by the X-server that invokes the
startxfce4 script. That's all the "login manager" does, although that's just a program being run by the display manager, not an actual "manager" like, for example,
kdm. (I.e, what do you hope to gain by not running that [minor] application?
)
If you want to avoid the "bother" of logging in, then just set your userid for automatic login.
