you want only members of group wheel to be able to su to root?
i've done this with pam once but i don't remember exactly how

you have to put something like this in /etc/pam.d/su: Auth required /lib/security/pam_wheel.so group=wheel
do a search in google for examples