I'm sorry, I'm not following you.
How much of this is true:
o You log in in terminal mode.
o You get to a (bash) terminal.
o You type startx& to bring up X.
o You press Ctrl-Alt-F1 to get back to terminal mode.
o You type killall X to get rid of X.
Don't you get back to your terminal prompt then? You should see the bash prompt again. Do you have to log in again? What Linux distribution are you using?