I wonder if it might be a glibc limit if the library was compiled with XOPEN2k defined:
grep -n NGROUPS_MAX -A3 -B3 /usr/include/bits/posix1_lim.h
76-/* Number of simultaneous supplementary group IDs per process. */
78:# define _POSIX_NGROUPS_MAX 8
80:# define _POSIX_NGROUPS_MAX 0
83-/* Number of files one process can have open at once. */
161-/* This value is a guaranteed minimum maximum.
162- The current maximum can be got from `sysconf'. */
165:# define NGROUPS_MAX 8
168-#endif /* bits/posix1_lim.h */
According to this mailing list response a side effect of increasing the value could eat up limited memory in the kernel. The message deals with NGROUPS_MAX in the kernel's limits.h header.
These limits of course are in the source which would mean recompiling.
What is the value of /proc/sys/ngroups_max?
Something in the back of my mind is telling me that using a different kernel version available might help, such as a 2.6 instead of 2.4 version kernel, or the memory page size the kernel uses. However, looking around on my laptop (64bit suse) everything points to 65536 max groups that can be assigned to a process. A search in a mailing list for rhel4 might reveal the answer.