You could try with getdtablesize() first to know the maximum number of file descriptors, then apply fcntl(2) on each one trying to get information, for example: the close-on-exec flag, discarding anyone that gives errors (errno should be EBADF).
The number that getdtablesize() gives is runtime, so it is the same as getrlimit(RLIMIT_NOFILE) or sysconf(_SC_OPEN_MAX). I managed to open file descriptors > 100 and then lower the limit with setrlimit() and keep them open, so it's best to grab a maximum value (which may be OPEN_MAX, usually present in <limits.h>).
I will share with you my code that puts all open file descriptors with the close-on-exec flag set on a select set. Changing it to return the number of open file descriptors is easy. Here's I'm using FD_SETSIZE as the upper limit.
Code:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/select.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
int get_ncloexec(fd_set *cloexec)
{
fd_set closet;
int flags;
int fd;
int n = 0;
FD_ZERO(&closet);
for (fd = 0; fd < (int) FD_SETSIZE; fd++) {
errno = 0;
flags = fcntl(fd, F_GETFD, 0);
if (flags == -1 && errno) {
if (errno != EBADF) {
#ifdef DEBUG
perror("fcntl(F_GETFD)");
#endif
return -1;
}
else
continue;
}
if (flags & FD_CLOEXEC) {
FD_SET(fd, &closet);
n++;
}
}
if (cloexec)
*cloexec = closet;
return n;
}