If you have nvidia hardware and are using their proprietary driver, you might want to try an older version - I was having similar issues recently whenever I launched a second Xserver. I was using 173.14.09 and then 169.12, both with 2.6.25 kernels, but I was forced to go back to 100.14.23 (and consequently a 2.6.24 kernel) before the problem went away. However I believe it was something in an xorg update that started triggering the issue because I was using 169.12 for some time with no issues before a few recent(somewhat) xorg updates.
Another possibility could be the evdev input driver. My mousewheel has left/right tilt buttons which for a long time did nothing at all on Linux. A couple months back hitting those tilt buttons (purely by accident) started crashing the Xserver so I went back to the standard "mouse" driver. That bug has been fixed now though, at least in Debian testing, and I'm using evdev again(with functional tilt buttons too).
If you use ATI hardware with their proprietary driver, try one of the open drivers for a while instead and see if the problem keeps occuring.
Apart from that evdev issue I was having, any Xserver instability I've ever had has been caused by ATI or Nvidia's proprietary drivers. Nvidia's is reasonably decent most of the time, but neither driver is particularly spectacular or completely stable imho.