Well, you can't get more restrictive without stopping your own access
By any chance, were you running gpg as a different user (root, for example)? From what I could find on google, running as a different user is the most common cause of that message.