Usually your 'run level' is set in the file /etc/inittab. It looks something like:
# Default runlevel. (Do not set to 0 or 6)
Depending on your distribution, you might have a 4 or 5 in there that signals to start x. If you can change this to a 3, save and reboot you will not start up x. If you can't edit this file, but are stuck in some hung x mode, try using the keystroke combination:
<CTL><ALT>F2, or <CTL><ALT>F3....up to <CTL><ALT>F7. This may bring up a console session in which you can log in and edit /etc/inittab or /etc/X11/XF86Config.