Assuming you use "startx" to get a gui, you can prevent X listening by editing startx. su to root then, using vi or your favourite editor, open /usr/X11R6/bin/startx
there's a line that says:
(or similar - not on linux box at the moment)
change it to read
You'll need to restart the x-server for it to stop listening (it'll happen next time you reboot if you don't want to faff about).
When you say X is listening, I assume you mean it's listening on the loopback (127.0.0.1 / localhost) rather than your actual network address?
i.e compare: nmap localhost with nmap <my_ip_address>