I'm not 100% sure, but I think I've had a similar problem about 6 months ago. It has something to do with permissions to connect to the X-server.
I've found in the file /etc/X11/xdm/xdm-config the following lines:
! SECURITY: do not listen for XDMCP or Chooser requests
! Comment out this line if you want to manage X terminals with xdm
! DisplayManager.requestPort: 0
After I've commented out the line (by putting an ! in front of it), I could connect using an X-terminal.