Problems restoring system from backup using rsync
 

Hi all. I have been using linux for about 2 years, but only seriously since early February. I have notes on how to backup in all different ways using tar and rsync, even using ssh with tar or rsync, doing incremental backup using find, rsync and ssh in one command, etc. I had my files backed up to several usb's, never really needed to restore anything.

Now I'm studying to be a RHCE and I know everything is a cake walk until you hit enterprise. I havent even hit the enterprise world and I'm already crawling. Definitely not walking and definitely not eating cake while walking (wherever cake came from LOL).

Anyway, I have a system here that I use for my playground. I corrupt it, re-install, corrupt, etc. I basically did a minimal install of centos, and then I tried to restore the system I am using right now, which is the same exact laptop and the same exact centos.

I used the command below
As you can tell, I disabled some files that I dont need backed-up, and did not include the system files. However, when I try to restore, I can't even log in. So then I did another fresh minimal install and only copied in this order, /etc, /home, /usr, so far so good but I went to another tty to log in as 'shawn' and it wouldn't let me. So then I got to copying /lib, and my system panicked and that was game over.

If someone can please explain how are you supposed to restore a system? I know about clonezilla but that isn't the point. I need to be primed for a production server, should some system just die, then what? Someone told me rsync and tar are powerful and better, faster, etc. I know there are 3rd party backup solutions, Im not even sure enterprise uses these, I wouldn't. I wouldn't want the chance of those getting attacked. I'd rather rsync incremental backups over ssh to a remote box in another building with super tight security on there. Root cannot even log in, etc.

Thanks!

You are excluding directories that need to exist with proper permissions. You don't need to copy the contents of them, but the directories need to exist.

You are right I dont include them when copying.
However, on the system with the minimal install, they are already there..

bump

I regularly do backup with rsync and restore to different computers. I use a file where I list what should be included or excluded. This is what I used for Arch. I'm not sure about /run, if it can be excluded in all distros. Some distros doesn't delete it at shutdown. I usually don't exclude it in Ubuntu, but then I exclude /run/media instead. (Or make sure nothing is mounted there.)

Then I run it as:
When I restore it I use a LiveCD. If I mount the root partition as /mnt the command is:
Edit: Added --delete, se my next post.

You can't exclude sda4, you have to exclude the mountpoint where it is mounted.

Ok thanks for responding. Just to be clear, "sda4" is a directory that i created from my /dev/sda4 partition. Sounds strange but I put Videos about linux and Music in there, then soft-link them to the directories under "Places" for easy browsing in Nautilus. So I excluded that directory.

I never used 'X' in rsync, but I read it preserves extended attributes, not sure what that means yet. I did not know I could use a file for all the includes/excludes. Great tip with that.

I just tried the setup you gave me and it did copy everything including hidden folders and directories. i was using 'rsync .*' from /. Not even sure what that did. Now I want to try restoring. I do like your method better. You copy proc and sys folders but not the contents. Smart. But either way, a newly installed system would have those files there. Is that where things get messed up? Should they be emptied out completely after doing a livecd rsync restore?

One thing I have experienced is that if you don't use X you are not allowed to ping as user when you have restored the backup. I don't know if there are other extended attributes used.

proc and sys should be emptied at shutdown.

I have never restored to a running system. If you restore from a Live CD then rsync will delete everything in proc and sys, if you use --delete. I forgot to mention that. Usually when I restore a backup I format the partition first. The only time I didn't, I messed it up because I forgot to use the delete-flag...

Makes sense on the --delete flag, but I see you totally format the partition first so that isn't necessary unless you didn't format.

I was trying to make a minimal or regular install first, and then copy the files over. I also didn't do it from a livecd either. Your method makes alot more sense, I mean, why would someone make a minimal install, and then restore an entire system when everything is already backed up?

Finally, what do you do when there are multiple partitions involved or even LVMs? Seems like a nightmare if trying to restore to another system with the same LVM setup or no LVM setup for the new machine. I guess you have to format the new machine exactly as the source, down to the cylinder. I guess that's why theres kickstart for new installs, then maybe do the rsync on a livecd using the --delete flag?

I guess a good question is, what is going on in a production environment? Since i am studying to become a RHCE. Do they use rsync and restore full systems from backup? I would think not, especially with how you mention little things go wrong like not being able to ping and who knows what else. If you make a kickstart install to 100 machines, how do you tweak the systems all to reflect all the security and the modified files for each client like LDAP or anything, ssh, dhcp, etc?

I guess an rsync restore doesn't seem like it would pan well in an enterprise environment but I don't know, I'm curious whats really going on out there.

I haven't used LVM so I know nothing about it.

If you want the restored system to have separate partitions for /home or /boot, just create the mountpoints and mount them before running rsync.

If the root partition is mounted at /mnt:

Yes that I understand already. I guess I wanted to know more about backing up from LVM to LVM or "migrating".

If you did do a kickstart install and you just wanted to transfer the data and config files over but not the packages and libraries (which really seem to mess things up at least for me, would copying /etc be ok or is there alot more to it than just /etc?

Arch has a wikipage about fullsystem backup with rsync There it says one should use -aAXv. Man rsync: "-A, --acls preserve ACLs (implies -p)" I don't know what that is...

I don't know about RHCE.

I use rsync to "install" (mostly) Xubuntu on a lot of machines, that anyway should be very much the same. Those machines are used as clients for surfing the web and whatever. So no servers on them. So the things I always change is fstab and grub to make them boot, and then hostname.

ACLs are Access Control Lists. I just studied that right now. Say a group called 'physics' own a directory called /physics and chmod everything 660 and the dir 770 so only physics can get in. But lets say they want to grant some user special priviledges to view a file or two in that directory but they aren't from the 'physics' group. They (or maybe just root himself) can grant access to that user to view a specific file in their directory and thats it with ACLs.

Yes what you are describing is for home use and works well. I do it too. Actually I just use clonezilla and it works like a charm. Never had an issue with it. only things that need changing are networking (host files, hostname, etc) and fstab and grub like you said. It doesn't copy proc and sys folders

So I guess rsync preserves these ACLs. Tar doesn't.

One thing I have thought about is the logs. It doesn't look like a freshly installed system if the logs are from several months back... (But it doesn't matter on those machines I maintain.)

I guess normally backup (with rsync or any other tool) is not used as a way to install a new system, but to restore the same system after a crash...

I was thinking the same thing. It can't be used in enterprise with too many variables going wrong. I think the only things necessary are the actual data files. Rsync is good for doing incremental backups. If you wanted to find only the files that have been changed in the last 24 hours in /home only and prune out the .mozilla directory and rsync them over SSH, here is the syntax..

So that would be good for small daily backups of the actual data but I would think other than data and configuration maintained in /etc, the rest of the system means nothing. It can always be re-installed.

Im new to enterprise but I'm guessing they probably do a bunch of kickstarts and install the necessary packages they need: LDAP, apache (if it isn't by default, can't remember) and post scripts. I would try to find a way to edit all the conf files in /etc on a massive scale so I wouldn't have to sit at 1000 machines doing that. Unless copying /etc to all of them would do just fine. I would assume

rsync is relatively slow for populating entire file systems because it writes individual files and checksums them. The quickest way to populate an entire file system is to use a tool that restores a complete file system such as clonezilla (as already mentioned) or dd to the partition's device file.

Ok I spent a couple days researching this whole thing and basically learned that for home use, clonezilla works great. Ive done, I use it for that purpose. In the enterprise world, it doesn't work that way. Rsync is not a system backup or restore tool. Its good for incremental backups every night for only modified files in the last 24 hours, or large files. If an enterprise system were to crash, you just take the important data that you backed up using incremental backups with rsync and copy the files over AFTER you do a fresh install with kickstart using a long post-script section to automatically edit every file for that system. This of course should be saved somewhere anyway.

a simple example would be:
echo "tty1" > /etc/securetty

Others can get more complex with sed and grep

The idea is early on when you set up the enterprise network, you manually set up 1 client machine. You save the kickstart file, then populate the rest of the kickstarts with their corresponding hostnames and IPs with another script. Put all the kickstarts on an NFS mount like '192.168.0.101-kickstart' so on and so forth. Then spend a whole day and tweak the machine manually and write down what changes were made. You can then either write out sed commands for each change to be pasted in the post-script section or copy the exact files on to the server and copy them over to all the machines in the network if that is easier. Its all in /etc anyway. The latter works if you are new to an already existing enterprise network and a system had crashed.

Regards "tweak the machine manually and write down what changes were made", one way is to back up every file before changing it with a rare extension, for example .org (short for original). When the tweaking is complete you can use find to do a context diff on all the .org files and store the output somewhere safe

If a restore is required the context diffs can be used to replicate all the changes.

Just wanted to say good point. The way I would do it would always be to create the backup file automatically with each file using the -i.bak option with sed as I would use sed for pretty much all my backups or at least I would use it first before I piped it into grep or awk.

Now that I would have a backup of all the files I created, I would want to know all the files I edited and have at least that information stored somewhere. So I would run a find command only on the files that have been modified in the last n minutes. Usually the post script section should take a couple minutes max so I could run a command to check for the files in /etc edited in the last 10 minutes or so and then send those files to the root directory of that machine for example or maybe to a remote machine, whatever I may desire...

But the diff you mentioned can work great to see the actual changes of the files which can be very useful. thanks for that note.