All,
So I have been tasked with trying to maintain the root password on close to 4000 Linux / Unix servers. My goal is to automate this process. As it currently stands, we basically have a team of admins that literally stop what they are doing to get the "root password update" done at the end of every quarter. This is obviously a problem. It usually takes about 3 weeks provided we have all hands on deck. I'm wondering if anyone has any suggestions as to what would work for my particular situation. I was going to write a Perl program that could do this for me however, If something already exists it would save me a lot of time. I have been researching this for the last few weeks and haven't really come up with anything that would work for my particular platform.
This is basically just a general question I'm hoping to get some feedback that will point me in the right direction. If you need anymore information let me know, I'll try to give as much as I can.
Thanks guys much appreciated!!
Orion.echo

My dad used to work for a large TELCO as a sys-admin, years ago, and he had a similar task, except he was the only person that could do it (all of the other sysadmins were busy with other tasks).

To accomplish his task, he used Expect - a Tcl superset. I hope that helps.

Same situation at the shop where I work. Groups of machines have root password set based on different criteria, thus root password is different based on the group the machine belongs to. Only thing that has saved us is the exchange of public keys for the root account. Yes, they still allow root ssh to systems. I've also looked into using a configuration management system, such as puppet, to accomplish the same. Expect is fine, but can be cumbersome if someone decides to change around root shell prompts. You need an agent that runs on all systems, as user root, that will accept remote authorized commands to update the password. Puppet will accomplish much of that.

Take a look for the fanout program (http://www.stearns.org/fanout/). Basically, you run this program from one computer, and it executes whatever you tell it, on multiple other systems, over SSH.

Expect has been mentioned, and is another good choice. Personally, if I had 4000+ servers to admin, I'd make sure I had LDAP installed and working.

Currently I'm using ssh keys for the root account as well. I will be disabling direct root log in very soon seeing that I'm having serious problems trying to find out "who" did "what" when something breaks. Also, I can't just pull root from everyone otherwise I would have done so (To many people complaining). My next step is to in fact disable direct direct root log in. This will allow me to be able to actually figure out who logged in and sudo'ed to root when something does break. Unfortunately, when I turn off direct root log in my keys won't be a means to get the password updated anymore. I also wrote an expect script that sort of works, and I find myself running into the exact same issue as what you mentioned tsandholm. We have so many different shell prompts my script starts to look very messy and hard to work with. I wonder if it would be possible to use cfengine? I read a little bit on what its purpose is but I really don't know for sure. I'm going to be researching more tonight on it.

Hmm.. Fanout, I have actually heard of this. It seems like its similar to cluster ssh. I will give it a test run. Also, I am actually working with another engineer regarding LDAP. I'm not to sure but I think hes trying to tie it in with Windows AD. Thanks for the info, I'm going to see what fanout has to offer.

4000 is a big number, if you had less I would have suggested you to use windows based software xshell. It has built in option to run a command on several systems simultaneously.

Obviously you would be fed up logging into 4000 different tabs . Anyways, if sometime you face a similar situation for around 50 odd machines, you could use it.

And as many people have already posted, expect is your true friend.

I stumbled across this article a while back http://magazine.redhat.com/2007/08/1...words-at-once/

It basically instructs you on how to create an RPM which securely changes the root password on install. Once the RPM is built and subsequent updates to the RPM are made, like a new password, it will automatically promulgate to round your servers.

One tip for the initial install is to add the new RPM as a dependency to something already installed on all the servers.

root account on LDAP??

I think fundamentally something is f*cked in the processes this company runs of they need to change passwords like this. on this scale, with any level of security, you can NOT use root passwords... ever. A dozen machines, fine, but without having things like crass templates to work out unique passwords, you can not manage them, therefore you should not use them. Change the root password to a very long and cryptic string (take the time, the hostname, and the serial number of the CPU / Hard drive and spit them all through md5 and use that for example), write it down, put in it an envelope, put it in a safe. repeat 4000 times... and never do it en mass again. That's why there are better frameworks like sudo which can sanely integrate on a per user basis into ldap and the likes. You have no business knowing any root password at all that way, and even less reason to ever need to use it, unless your ldap infrastructure crashes and burns or something equally horrible.

True....I'd not put root under LDAP either. Root under LDAP is a bad idea, and I probably wouldn't have suggested it, if I wasn't so tired yesterday.

I'm 100% aware that this is insecure, I'm trying to fix this. Unfortunately it's not as easy as you might think. I can't just strip the root password from everyone. There are a lot of "higher ups" that won't allow this so I need to take baby steps to get a new process implemented. I have anywhere from 25 to 1000 users so getting the sudoers file fixed is also going to take time. The idea behind turning off direct root login's will allow me to get more info on who needs access to what server. Once this access is pulled I will be getting flooded with emails regarding adding / fixing user accounts. This is great, that means that I will be able to fix our sudoers file. I completely agree with you this is ridiculously insecure but its a legacy process so I have to make small changes verify that nothing breaks then move on. I was the one that complained about it being insecure so of course I get stuck with trying to fix the problem. The unknown root password is my ultimate goal but it's not going to happen over night.

Anyone who's inherited a system will know what you mean and understand. I find that the smaller someone's experience of real life commercial implementations the less willing they are to appreciate sub-standard solutions can be unavoidable, instead doggedly sticking to their personal recommendations regardless. Basically... I know the feeling.

Well, I appreciate all the feed back. Looks like I'm going to re-engineer my expect script.
Thanks guys!

I would not allow remote login for root, and instead I assign sudo permissions for some normal account, so that they can act as root.
Some password policy can be applied on these normal account, so they are meant to have change their password when needed.

This seems to me to be an interesting technical problem, from the organizational perspective. How do you securely keep track of all of the password requests?

If you put together a fairly simple secure web site, feeding in to a database, you could make your job much easier: username, server host name, and a few other fields of your choice can be validated by a web server, rather than given as free form text in email.

Then you set up a second page for yourself, populated by the database, of everyone who wants access, with a series of check boxes. Check the boxes of those people you want to grant access to, click submit, and automatically augment the sudoers file with those users. By the time that you've answered your first 500 emails, you've probably bought yourself back all the time that it took to put the web site together.

When you're finished with the web site, you send out an all company email telling people that their requests for an entry into the sudoers file will be processed much more quickly if people click on the link in the email, rather than by sending email to the IT department.

Off topic, I realize, but could be useful.