What are you actually trying to accomplish by limiting the number of users allowed to login? Wouldn't you be better off limiting what users are allowed to do instead of limiting the amount of users. It only takes one user to bring a system to a crawl by stealing all the resources.
Have you looked at the possibilities of PAM? If never really got into using it but it has lots of different possibilities of limiting access.
If you can't find anything with PAM you could always write a simple script which disables users from logging in.
Code:
if [ `w -h | wc -l` -gt 10 ]
then
echo "Too many users" > /etc/nologin
else
rm -f /etc/nologin
fi
You could run that from cron or have it triggered when a new user logs in.