You're still running X in that scenario, you just don't have
a window-manager ...
The "trick" is to have e.g. a ~/.xinitrc with e.g.
as it's last (and olny line with an exec) line.
That will bring up X with only one program (emacs)
If you're using a graphical login-manager, make an
analogous .xsession file ...
Alternatively, you can mess around with xinit,
but I don't know a "proper call" off the top of my
head, have a read ;)