Welcome to LQ.org, SweetPete!
Yes, if you don't need the GUI, definitely go without it. Judging by the hit on the CPU and size of the XFree86 package, I'd say you're effectively doubling the system overhead running X Windows.
Besides
serious Linux/Unix users don't use GUIs.
Bert