I know I'm replying to an old post... but has anyone solved this one?
My RH8 box (dedicated FTP server) was exhibiting exactly the same symptoms you've described. I replaced \etc\group and \etc\group- with known good files from an identical box, and then I stopped getting the error. However, I don't believe I've reached the root of the problem yet.
Every time I add a user, I start getting the error again. I can boot to the console and issue "startx", and X will run, but the error (as you described) annoyingly pops up about every 30 seconds.
While booting I'm still seeing an error, which I suppose is related:
chgrp: invalid group name "utmp"
I was getting this error before replacing the "group" files as noted above.
Since an FTP server is a relatively easy mark for intruders, I'm beginning to wonder if my xf86 security has been compromised from the outside?