Scenario - primary drive crashes, have 2nd drive with full backup of all files (using cp -a on weekly basis). Datacenter restores minimal OS on new primary drive.

Is there a way to get the server back to its state at the time the last backup was taken? I know I can "cp -af" /home, /var, /etc, /root, /sbin, /usr, but what about yum/up2date updates that affected /lib and /bin? Trying to copy over /lib always crashes the server.

I know I could update the OS, recompile sources, etc... and then restore user data and configs, but I'm looking for a way to easily restore from a backup drive that has a full copy of the server prior to the crash, all without console access. My problems seem to be in restoring /lib and /bin to their "pre-crash" state to make sure all binaries and libraries match.

Rob

If you are installing from source and putting your files into the "normal" hierarchy, then that's an error. You should be putting source compiled software into /usr/local/ or /opt. Most source installation scripts do this as the default behavior.

This allows you to upgrade your system software using the standard distribution tools (apt, yum), and to copy back any non-standard software from backup, along with your user data.

Be careful when copying back /var - that hierarchy includes the system software database for many systems. Overwrite that, and you'll need to re-install again to recover.

Thanks for the reply. Our source installations are going into /usr/local or /usr, so those should be fine. So, I assume then that after the OS restore, we should run up2date/yum, upgrade the kernel to the version that was on the backup, and then restore from backup /root, /home, /etc, /usr, /var ? /var should be ok to "cp" over as it was a full backup I would think.

I guess my concern is if we did not re-install all the previous yum/up2date packages, then /var rpm db's will be out of sync with /lib and /bin? Should our backup process also include a file of the output of "yum list installed" and then do a "yum update <list>" after the restore?

Just trying to come up with all the potential snags/pitfalls so when the "crash" occurs, we have a plan that makes recovery quick and straightforward.

Rob

The directories /usr and /var are not OK to overwrite. The maintenance you apply today will not be identical to the maintenance at the time of your backup. By overwriting /usr and /var, you have effectively trashed the integrity of your system software and rpm database.

I usually update a copy of the output of 'rpm -qa' before the backup. This is essentially the same as a "yum list installed", in a different format. After reinstalling and updating the system, I add the third party repositories I was using. I then build a yum install for the missing packages (from a diff of the 'rpm -qa' files) with an ad-hoc script, which installs them from the third party repositories. The only additional software will be reinstalled by the restore of /usr/local on my systems. Update the /etc and /var/named (for DNS) files, add the users and their data, and the system is back.

I run this process each time I upgrade the system to a new release (most recently FC5->FC6). I use machines with i686, x86_64 and PPC architectures, and the process is the same.

I keep a 'warm backup' for the servers; a machine that the server rsyncs live configuration to several times a day. If the primary server explodes, the warm backup can be brought online in a few seconds. This gives you the luxury of time to restore the primary server carefully.

Thanks for the additional details. I have a clearer understanding of what's involved now. My restore process is now:
- yum/up2date diff of rpm -qa
- update kernel
- re-install cpan, cpan update perl modules
- re-install razor, spamassassin (from source, as they go in /usr)
- copy (cp -af) from backup:
/home, /etc, /root, /usr/local, /var/spool, /var/log, /var/named, /var/tmp, /usr/share/ssl, and a few custom perl modules in the path /usr/lib/perl5...
- reboot

The above seems to cover all the things I can think of. If you see any problems, see a way to accomplish this better, or anything else you think I missed, please post.

Rob