I'm thinking more about OS patching here but you state that there are implications for the production service so some of the following may be relevant.
Are all the 1000 servers identical; models, memory, hbas, nics, processors, etc? If they're all the same, are the firmware levels of BIOS, Array controllers, HBAs, nics and any other adapters all at their highest level? I doubt it. They'd all also have to be running exactly the same version of OS and the same applications, at the same rev level. OK, so they'd just be some of the things you need to be aware of. You can't always have the most current firmware or OS Rev if your running some old legacy version of Oracle or something.
To upgrade production servers you'd first need to try upgrading test server(s) which were the closest match to the production ones. (You may have to group your production ones by type, hardware, OS rev, application, etc, and upgrade several test ones this is all to prove the package has no detrimental effects.)
If the test servers upgrade goes OK then you think about upgrading the DR rather than the production ones. Even assuming they were all identical in all respects, I can't really see you running ONE script to do 1000 servers, you're going to have to plan how to do them some at a time, and I'm talking low numbers. You must
always have a backout path available if/when it all goes wrong.
Meanwhile, back at the upgraded DR servers... once they're upgraded and are deemed stable, you fail the production servers over to the DR servers (You'll be doing these one at a time...) Should the upgrade fail (Production is now on the DR servers) you can fail back to the original production servers with minimum loss of service.
In most large commercial situations you'll also find that various business groups "own" the servers so they have to be consulted to authorise the work. you'll also need to get resources on side; Oracle DBAs, App analysts, etc, storage people, network people, etc, if the deployment involves any sort of reboot or affects anything within their remit.
That lot may bear absolutely no relation to what you're trying to do but 1,000 servers says to me that you're talking a reasonably sized business which doesn't want to find itself with some major IT problem because that many servers got screwed during a package deployment.
When you plan this,
ensure that you have a backout solution which will cause minimum downtime should it all go horribly wrong!
Lastly... Please tell me you don't work for a Bank!!
Play Bonny!
