This very interesting. You might have already done this, if not please place account required pam_time.so as the first line of /etc/pam.d/login.
The next thing to notice is your login logs. If you check /var/log/secure after a test login and log-off, you will see an entry such as pam_unix(sshd:session)... or pam_unix(gdm:session)... based on your login method.
If you see these add an additional line to/etc/pam.d/sshd session required pam_time.so. Now do the same to /etc/pam.d/gdm if you need to restric GUI access.
Unfortunately this is the drawback of PAM's modular approach.