I have had this problem before
The problem seems to be the glibc.
For the longest time I thought it was just me having that problem. (and it is exactly how you explained it, I use xfce)
I believe, the 2.3.6 is the most stable (of the glibc's above 2.3.3), except nobody can ssh the system. (as far as my tests go)
glibc-2.3.5 is the most unstable. If your using that please change to 2.3.4 or 2.3.6.
---
Although the possibility exists there is something we both did during compliation.. somewhere.. that caused xfce4 to have the crash, when I changed only the glibc and nothing else, the crash you speak of vanished. (and glibc-2.3.5 introduced MORE crashes...)
I used the fully loaded installation of xfce4.
Your problem, however similar, may be different that mine. If they give a suggestion try theirs first as it is less...substantial of a change. (and I cannot guaruntee that I did not personally screw up somewhere quite badly..)