Here's an example of how to test the pam_tally mechanism. I'll be using a CentOS-3 box w/o SELinux (don't affect this anyway AFAIK), using the PAM service "su", local user "test" and default faillog records in /var/log/faillog. * Note no system should have a user called "test" unless you have a) made it an unprivileged user and b) set a ridiculously strong password and c) restricted access to it in the most BOFH-stylee kind of way.
0. Check if you got /var/log/faillog: using pam_tally's "onerr=fail" will cause a DoS if the file doesn't exsist, so make sure you want that. Most likely you'll want "onerr=succeed" and let another module in the PAM stack deny or allow access to the service.
1. Check for user "test" if it's got a non-zero ".fail_max" in faillog: "faillog -u test" (or "pam_tally --user test"). Enter a maximum or 2: "faillog -u test -m 2" (and check it's set correctly).
2. Edit your /etc/pam.d/su stack:
auth sufficient /lib/security/$ISA/pam_rootok.so
auth required /lib/security/$ISA/pam_tally.so onerr=succeed
auth required /lib/security/$ISA/pam_stack.so service=system-auth
account required /lib/security/$ISA/pam_tally.so deny=3
account required /lib/security/$ISA/pam_stack.so service=system-auth
password required /lib/security/$ISA/pam_stack.so service=system-auth
session required /lib/security/$ISA/pam_stack.so service=system-auth
** The "deny=3" is only used for users that don't have ".fail_max" in faillog.
3. Now try to su to user "test" and give a vronk password. Repeat once more. Check with faillog: "faillog -u test" or "pam_tally --user test" and see the counter get incremented.
4. Now try to su to user "test" giving the correct password and see access being denied.
Please make it a standard procedure to first use the faillog / pam_tally tools and read the system logs to check for errors before reporting failures.
*** Cracklib is for password strength, not denying logins, so if you say "retry=3" you're giving 3 *chances* before returning. A user can rerun it w/o probs, no denial. And if you use "remember" make sure you have got a file "/etc/security/opasswd" for storing old passes (if the RPM didn't create this already).