I am using RHEL 5 Server and trying to setup the /etc/pam.d/system-auth file so when users login on the workstation there account is locked after 3 failed attempts.

Using the NSA guide I performed the below:
changed below lines -

password requisite pam_cracklib.so try_first_pass retry=3
TO
password required pam_cracklib.so try_first_pass retry=3 minlen=8 \
dcredit=-1 ucred=-1 ocredit=-1 lcredit=0

changed below lines -
auth sufficient pam_unix.so nullok try_first_pass
TO
auth required pam_unix.so nullok try_first_pass

deleted below lines -
auth requisite pam_succeed_if.so uid >=500 quiet
auth required pam_deny.so

added below lines -
auth required pam_tally2.so deny=5 onerr=fail

added below lines -
account required pam_tally2.so

The account lockout feature (the screens nofied the user the account is locked) so it works but my root keeps getting locked out so I have to boot to the CD-ROM and recover the file. How can I prevent my root from getting locked?

Read here:
/usr/share/doc/pam-0.99.6.2/txts/README.pam_tally2

And let's see your entire /etc/pam.d/system-auth file contents (in code tags). It's not supposed to work the way you are describing. You could set magic_root but that shouldn't be necessary...

The reference to /usr/share/doc/pam-0.99.6.2/txts/README.pam_tally2
is awsome! What I found is that the order in which you enter your options is critical. Apparently you must add your options in the sequence outline in the file README.pam_tally2. Also I found there is no need to add anything to ensure root is NOT locked out. I have managed so far to lock out an account for a period of 5 minutes and it was reset automatically. I also attempted to force root to lockout and so far have not been able to at the console(which is good). I did not have to type any special comments to ensure root did not lockout.

Advice to new users who read the attached document is when in a specific category, like "auth" enter your options and variables in the sequence outlined in this document. Example:

auth required pam_tally2.so [I]deny=5 unlock_time=300

Don't try pam_tally2.so unlock_time=300 deny=5

A million thanks anomie

Take care,
Johnny Mac

In my post above a [i] got added so ignore where you see the [i] as seen in full below:

auth required pam_tally2.so [i]deny=5 unlock_time=300