Hi all,
I have a general question about the server administration. When I first started we used to use one application per server but now with virtualization like VMware we are implementing multiple servers with multiple applications in one single hardware. Like for me I used to handle around 40 servers but now due to disaster recovery we implemented DR site and replicate all the production servers there in VMWare which just doubled the server count for me and it's rising.
It's quite impossible if it rises this way to remember just the root password of all those servers. For monitoring may be we can use nagios/cacti other tools to automate the monitoring process but how to manage these servers in large organizations; they just segregate the servers between admins? But still the count will eventually get high one time. Like say for an update of kernel patch how to do that in all those servers, one by one? Last year we had an issue to update the timezone for DST and I had to login in each server and do that which was really painful.
So whats your professional experience regarding this.

If it is Redhat/fedora/Centos. Please search for func (Fedora unified netowork controller). Func is a tool to control many servers from single location.

perhaps you can use Puppet or Cfengine

I guess a 'high number' of servers is relative term. In the world if HPC, managing a cluster of a few thousand servers is now commonplace, but these clusters all have the same property - either all the servers are identical or there is a relatively small pool of common images. They may run different apps, but the o/s image/configuration on them is the same. I don't know if this applies to your situation or not but it greatly simplifies things.

When changes are made, they're made to an image and than pushed out to all servers intended to run that image.

Another technique that might help you is naming. All servers had a common naming format as well as simplified aliases so you might have names like: n1 n2 ... n1000 or maybe there's a rack number or other discriminator in there like: r1n1 r1n2... r2n1 nr2n2...

Of course having ssh-less access for the management accounts is mandatory. Then you get access to 'pdsh', which stands for parallel distributed ssh. Let's say you want to install an rpm named foo on 1K servers! Piece of cake. First you copy it with the distributed copy utility:

pdcp -w n[1-1000] foo.rpm /tmp

and then install it on 1000 servers

pdsh -w n[1-1000] rpm -ivh /tmp/foo.rpm

Virtually any operation that needs to be done multiple times can be done this way. Even if your servers don't have common names you might be able to add aliases to your hosts files to make navigation more consistent. in fact, even if you can't do that you can always put the names in a file and point pdsh at it. Of course having password-less ssh is a must. If you don't have that you should ask for a raise.

-mark