I have had this problem before
The problem seems to be the glibc.
For the longest time I thought it was just me having that problem. (and it is exactly how you explained it, I use xfce)
I believe, the 2.3.6 is the most stable (of the glibc's above 2.3.3), except nobody can ssh the system. (as far as my tests go)
glibc-2.3.5 is the most unstable. If your using that please change to 2.3.4 or 2.3.6.
Although the possibility exists there is something we both did during compliation.. somewhere.. that caused xfce4 to have the crash, when I changed only the glibc and nothing else, the crash you speak of vanished. (and glibc-2.3.5 introduced MORE crashes...)
I used the fully loaded installation of xfce4.
Your problem, however similar, may be different that mine. If they give a suggestion try theirs first as it is less...substantial of a change. (and I cannot guaruntee that I did not personally screw up somewhere quite badly..)