If I understand you correctly, you've done it the wrong way round. You need to add user N to each of the user's groups, not the way you've done it (which was adding each user to group N).
However, even this isn't ideal, since it gives N access to most of the other files created by any other users, (unless you have some sort of clever umask settings). Plus it gets unwieldy if there's lots of users. A better way to do it would be with POSIX ACLs. Have a look
here. It's old (FC2), but still a good introduction to ACLs in linux.