The problem is probably being caused by gdm. There is a line in gdm.conf that will probably look like this:
This is blocking normal TCP from getting in. SSH works because the TCP stuff is tunneled through SSH.
You may also find this line in /usr/share/gdm/defaults.conf. A lot depends on the distribution.