passwd: Authentication token manipulation error message

kaplan71 · 06-14-2011, 09:50 AM

Hi there --

We are running Fedora Core 14 64-bit, and users who have accounts on the server are having problems changing their respective passwords via the passwd command. The command fails with the following error message:

Quote:

passwd: Authentication token manipulation error

Significantly this message appears on-screen after a few seconds of either activity or no activity.

I have already recreated the /etc/shadow file via the pwconv command syntax. Even after I had done this, the problem has continued to occur. The permissions on the /usr/bin/passwd file appear to be correct including the sticky bit being present:

-rwsr-xr-x 1 root root 27936 Aug 11 2010 passwd

Is this a case where the passwd binary is corrupt, and needs to be restored from backup, or is there something more fundamental going on here?

Thanks.

anomie · 06-14-2011, 12:38 PM

Do you recall doing any PAM tinkering before the problem started occurring? If so, that would be the first place to check (/etc/pam.d/passwd and/or /etc/pam.d/system-auth on Fedora systems).

If that yields no promising leads, then use strace(1) to observe where the passwd(1) program is falling down.

BTW, it would be strange for a binary to become "corrupt" on a Linux system in the way (I think) you're thinking. Such an event would probably be preceded by lots of noisy disk errors and a system that had difficulty booting at all.

kaplan71 · 06-14-2011, 01:00 PM

Hi there --

I checked the /etc/pam.d/system-auth file, and I vaguely remember doing some tinkering to it. I don't remember what the default settings were for the file. The current text of the file is shown below:

Code:

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3
password    required      pam_unix.so md5 remember=12 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

As far as the /etc/pam.d/passwd file is concerned, that does not appear to have been changed in any way. Its context is the following:

Code:

%PAM-1.0
auth       include      system-auth
account    include      system-auth
password   include      system-auth

The problem is probably with the system-auth file, and it should probably be put back to its default setting. The question is, what is the correct syntax for it? One other thing, I was mistaken in the OS version of the server. It is the CentOS 5.6 distribution that is on the server.

Thanks, and sorry about the misinformation.

anomie · 06-14-2011, 03:05 PM

Take the remember=12 directive out of that line (i.e. remove that key/value pair, not the whole line). See if it works afterward.

On CentOS 5, per /usr/share/doc/pam-0.99.6.2/txts/README.pam_unix, the remember=n directive should work, though. Maybe it wasn't implemented correctly - assuming that is the problem.

Another note: when you're seeing PAM problems, two good places to start checking for clues are /var/log/secure and /var/log/messages.

kaplan71 · 06-14-2011, 03:24 PM

Hi there --

I modified the line in question to read as follows:

Code:

password    required      pam_unix.so md5 shadow nullok try_first_pass use_authtok

I tried changing the password of a user's account, and the error message occurred again.

I then modified the line to read as shown below:

Code:

password    required      pam_unix.so

I repeated the same passwd command syntax as before, but the error message continued to occur. I have put the original syntax back into the line.

I checked the /var/log/secure and /var/log/messages files for any entries related to the above steps, but there were
none that were apparent.

Any other ideas?

anomie · 06-14-2011, 03:47 PM

So it may be that my remember=n directive idea is just a red herring. Probably safe to ignore that advice.

Next step: observe the system calls using strace(1) to determine where the problem is occurring.

kaplan71 · 06-14-2011, 05:04 PM

Hi there --

I ran the strace passwd command syntax for one of our user accounts, ahk, and an excerpt from it is shown below:

Code:

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ad50d13d000
read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 4013
close(4)                                = 0
munmap(0x2ad50d13d000, 4096)            = 0
open("/etc/passwd", O_RDONLY)           = 4
fcntl(4, F_GETFD)                       = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=4013, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ad50d13d000
read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 4013
close(4)                                = 0
munmap(0x2ad50d13d000, 4096)            = 0
open("/etc/shadow", O_RDONLY)           = -1 EACCES (Permission denied)
write(1, "Changing password for ahk\n", 26Changing password for ahk
) = 26
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
rt_sigprocmask(SIG_BLOCK, [INT TSTP], [], 8) = 0
ioctl(0, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost isig icanon -echo ...}) = 0
write(2, "(current) UNIX password: ", 25(current) UNIX password: ) = 25

One thing that I noticed was the line that read as follows:

Code:

open("/etc/shadow", O_RDONLY)           = -1 EACCES (Permission denied)

I checked the permissions on /etc/shadow, and here is the readout:

Code:

-r--------   1 root root    3313 Jun 14 14:20 shadow

I checked the secure and messages log files, but there was nothing in either that would indicate a problem.

anomie · 06-14-2011, 11:09 PM

Can root change his password?

The 'permission denied' message would imply the suid bit on passwd(1) is not working properly. Have you changed anything else recently that you can think of? (Filesystem mount options, perhaps?)

Reuti · 06-15-2011, 04:34 AM

I would suggest to make shadow writable for root.

kaplan71 · 06-15-2011, 08:55 AM

Hi there --

Thanks for the replies. The answers to the questions are the following:

Quote:

Can root change his password?

The root account cannot change its password. The same error message occurs after a few seconds of either keyboard input or no keyboard input.

Quote:

Have you changed anything else recently that you can think of? (Filesystem mount options, perhaps?)

There have been no changes that I am aware of that were made to the filesystems. The context of the fstab file is shown below:

Quote:

/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0

Quote:

I would suggest to make shadow writable for root.

I ran the chmod 600 command on the shadow file, and tried changing the root account's password. The same error message continued to occur. Because the change did not help matters, I reran the chmod command, and reverted the permissions back
to the 400 status for the shadow file.

anomie · 06-15-2011, 09:57 AM

I'm running short on ideas this morning. If you can afford to schedule a downtime, I'd reboot the system to single-user mode and force a fsck(8) on /dev/VolGroup00/LogVol00.

Reuti · 06-15-2011, 09:59 AM

Did you try changing the root password and setting shadow 600 at the same time?

It’s also mounted as rw according to the output of mount?

The passwd binary needs the suid bit set to work for ordinary users, and so the file system mustn’t be mounted with nosuid.

Do you run AppArmor on the system?

kaplan71 · 06-15-2011, 01:53 PM

Hi there --

Thanks for everyone's suggestions. I was able to solve the problem after trying several things.

I first changed the system to single-user mode via the init 1 command. Once there, I ran the command:

Code:

fsck /dev/VolGroup00/LogVol00

and corrected a series of outstanding errors on the filesytem. After that, the system was returned to runlevel 5 via the init 5 command. The error message continued to occur.

I then tried to run the passwd command to change the root account's password while changing shadow to read/write status for root using the syntax:

Code:

# chmod 600 /etc/shadow && passwd

This attempt was also unsuccessful.

After that, I decided to do a "generic" install of the operating system on another computer. Once that was done, I made a copy of the /etc/pam.d/system-auth-ac file on the problem computer, and copied over a "fresh" version of the file from the second computer.

The replacement file appeared to do the trick, because when I ran the passwd command after the replacement file was in place, I was able to change the root account's password without any error.

Reuti · 06-15-2011, 02:05 PM

Can you tell us what’s inside the replacement file or what’s the difference, so that we can learn from it?

anomie · 06-15-2011, 02:42 PM

Yes, we'd like to see the borked system-auth-ac if you kept a copy (or if you can pull one from backups). I'm a little surprised passwd(1)'s PAM calls were even reading it. I did not see reference to it in the stacks you posted.