What is the best practice for dist-upgrading a very large production server?
I've been given a server running Ubuntu 12.04 LTS and I would like to bring it up to at least the next version 14.04 LTS, or preferably the current version 18.04 LTS. The server has ~ 32 TB of data spread out over several Linux volume groups mounted to the filesystem, with /var taking up nearly 10 TB and backups from other servers being kept on /mnt and consuming 22 TB. It's a bare metal server, not a virtual machine, and it's also in production so the upgrade process needs to go smoothly so that our code repository and also our ticketing system remain available with minimal downtime. Removing the backups mounted to the filesystem would trim the overall filesystem down to ~ 10 TB. Most of our applications and the usual system stuff are resident in /var however so I cannot simply unmount it in order to shrink the filesystem any further. The LVGs are carved out of a single RAID 5 array. (I'm aware that RAID 6 or RAID 10 would be preferable but I was recently handed the machine and did not design or build it.)
Ideally, when doing an upgrade to a server, I like to duplicate the environment as closely as possible and test the process in order to make sure that there's no issue with either upgrading the kernel or especially the applications. In this particular case, I don't have TBs of space anywhere to either clone the machine for a VM or have the resources to create a duplicate bare metal server. As 12.04 has now reached EOL, and the server has not had security updates in a long time, it is imperative to upgrade the server as soon as possible to at least 14.04 LTS. I've done *nix upgrades before in my career, but I'm not a Linux specialist, and I don't have any experience with a server this large.
I've googled for the best practices for testing and dist-upgrading a large production server but I'm not getting anything particularly useful. Does anyone have any resources that they could point me to or have any advice for the best way to tackle this beast? Thank you in advance for your assistance.
|