How can we deploy package on 1000 servers
 

how to deploy a package change to 1000 servers. The change will alter the way that the production service operates. What precautions would you take to ensure that this deployment has minimum affect on service performance and availability

Hi...

While I have no experience in this particular area, I do know that you will need to provide more information for the benefit of those who do. Please start by reading here and telling us what version of Linux you're using and what packages you are wanting to install or change. ;)

Regards...

I'm thinking more about OS patching here but you state that there are implications for the production service so some of the following may be relevant.

Are all the 1000 servers identical; models, memory, hbas, nics, processors, etc? If they're all the same, are the firmware levels of BIOS, Array controllers, HBAs, nics and any other adapters all at their highest level? I doubt it. They'd all also have to be running exactly the same version of OS and the same applications, at the same rev level. OK, so they'd just be some of the things you need to be aware of. You can't always have the most current firmware or OS Rev if your running some old legacy version of Oracle or something.

To upgrade production servers you'd first need to try upgrading test server(s) which were the closest match to the production ones. (You may have to group your production ones by type, hardware, OS rev, application, etc, and upgrade several test ones this is all to prove the package has no detrimental effects.)

If the test servers upgrade goes OK then you think about upgrading the DR rather than the production ones. Even assuming they were all identical in all respects, I can't really see you running ONE script to do 1000 servers, you're going to have to plan how to do them some at a time, and I'm talking low numbers. You must always have a backout path available if/when it all goes wrong.

Meanwhile, back at the upgraded DR servers... once they're upgraded and are deemed stable, you fail the production servers over to the DR servers (You'll be doing these one at a time...) Should the upgrade fail (Production is now on the DR servers) you can fail back to the original production servers with minimum loss of service.

In most large commercial situations you'll also find that various business groups "own" the servers so they have to be consulted to authorise the work. you'll also need to get resources on side; Oracle DBAs, App analysts, etc, storage people, network people, etc, if the deployment involves any sort of reboot or affects anything within their remit.

That lot may bear absolutely no relation to what you're trying to do but 1,000 servers says to me that you're talking a reasonably sized business which doesn't want to find itself with some major IT problem because that many servers got screwed during a package deployment.

When you plan this, ensure that you have a backout solution which will cause minimum downtime should it all go horribly wrong!

Lastly... Please tell me you don't work for a Bank!! :)

Play Bonny!

:hattip: