Jefro makes a sensible point.
If you want run X (maybe you want to be able to start GUI programs from the terminal) but just use xterm, install and use TWM
, the simplest window manager of all. All it does is start xterms. I would expect it to be in the repos, though I have no way of easily checking.
If you want to boot directly to the command line, that will be another story. I set up my Debian box to do it but never attempted it with Ubuntu, though there most certainly should be a way.