LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 11-10-2012, 12:29 AM   #1
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Rep: Reputation: Disabled
Problems restoring system from backup using rsync


Hi all. I have been using linux for about 2 years, but only seriously since early February. I have notes on how to backup in all different ways using tar and rsync, even using ssh with tar or rsync, doing incremental backup using find, rsync and ssh in one command, etc. I had my files backed up to several usb's, never really needed to restore anything.

Now I'm studying to be a RHCE and I know everything is a cake walk until you hit enterprise. I havent even hit the enterprise world and I'm already crawling. Definitely not walking and definitely not eating cake while walking (wherever cake came from LOL).

Anyway, I have a system here that I use for my playground. I corrupt it, re-install, corrupt, etc. I basically did a minimal install of centos, and then I tried to restore the system I am using right now, which is the same exact laptop and the same exact centos.

I used the command below
Code:
# cd /; rsync -av .* /media/centos-backup --include=dev/console --include=dev/initctl --include=dev/null --include=dev/zero --exclude=home/shawn/Videos --exclude=home/shawn/Downloads --exclude=mnt --exclude=media --exclude=proc --exclude=sys --exclude=var/spool/squid --exclude=dev/* --exclude=home/shawn/.mozilla/firefox/*/Cache --exclude=tmp --exclude=lost+found --exclude=sda4 --exclude=home/shawn/.thumbnails --exclude=videos --no-group
As you can tell, I disabled some files that I dont need backed-up, and did not include the system files. However, when I try to restore, I can't even log in. So then I did another fresh minimal install and only copied in this order, /etc, /home, /usr, so far so good but I went to another tty to log in as 'shawn' and it wouldn't let me. So then I got to copying /lib, and my system panicked and that was game over.

If someone can please explain how are you supposed to restore a system? I know about clonezilla but that isn't the point. I need to be primed for a production server, should some system just die, then what? Someone told me rsync and tar are powerful and better, faster, etc. I know there are 3rd party backup solutions, Im not even sure enterprise uses these, I wouldn't. I wouldn't want the chance of those getting attacked. I'd rather rsync incremental backups over ssh to a remote box in another building with super tight security on there. Root cannot even log in, etc.

Thanks!

Last edited by rootaccess; 11-10-2012 at 12:36 AM.
 
Old 11-10-2012, 07:33 AM   #2
andrewthomas
Senior Member
 
Registered: May 2010
Location: Chicago Metro
Distribution: Arch, Gentoo, Slackware
Posts: 1,690

Rep: Reputation: 307Reputation: 307Reputation: 307Reputation: 307
Quote:
Originally Posted by rootaccess View Post
I used the command below
Code:
 --exclude=proc --exclude=sys
You are excluding directories that need to exist with proper permissions. You don't need to copy the contents of them, but the directories need to exist.
 
Old 11-10-2012, 02:07 PM   #3
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by andrewthomas View Post
You are excluding directories that need to exist with proper permissions. You don't need to copy the contents of them, but the directories need to exist.
You are right I dont include them when copying.
However, on the system with the minimal install, they are already there..
 
Old 11-12-2012, 09:47 PM   #4
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
bump
 
Old 11-14-2012, 04:25 PM   #5
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 302

Rep: Reputation: 61
I regularly do backup with rsync and restore to different computers. I use a file where I list what should be included or excluded. This is what I used for Arch. I'm not sure about /run, if it can be excluded in all distros. Some distros doesn't delete it at shutdown. I usually don't exclude it in Ubuntu, but then I exclude /run/media instead. (Or make sure nothing is mounted there.)

Code:
# backup.lst

# Include
+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero

# Exclude
- /dev/*
- /proc/*
- /sys/*
- /tmp/*
- lost+found/
- Trash/*
- /mnt/*
- /media/*
- /run/*
- /home/*/.gvfs
- /home/*/.macromedia/*
- /home/*/.mozilla/firefox/*.default/Cache/*
- /home/*/.thumbnails/*
Then I run it as:
Code:
# rsync -aXv --delete-excluded --exclude-from=/path/to/backup.lst / /media/backup/
When I restore it I use a LiveCD. If I mount the root partition as /mnt the command is:
Code:
# rsync -aXv --delete /media/backup/ /mnt/
Edit: Added --delete, se my next post.

You can't exclude sda4, you have to exclude the mountpoint where it is mounted.

Last edited by Lennie; 11-17-2012 at 05:21 AM.
 
Old 11-17-2012, 04:53 AM   #6
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
Ok thanks for responding. Just to be clear, "sda4" is a directory that i created from my /dev/sda4 partition. Sounds strange but I put Videos about linux and Music in there, then soft-link them to the directories under "Places" for easy browsing in Nautilus. So I excluded that directory.

I never used 'X' in rsync, but I read it preserves extended attributes, not sure what that means yet. I did not know I could use a file for all the includes/excludes. Great tip with that.

I just tried the setup you gave me and it did copy everything including hidden folders and directories. i was using 'rsync .*' from /. Not even sure what that did. Now I want to try restoring. I do like your method better. You copy proc and sys folders but not the contents. Smart. But either way, a newly installed system would have those files there. Is that where things get messed up? Should they be emptied out completely after doing a livecd rsync restore?
 
Old 11-17-2012, 05:15 AM   #7
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 302

Rep: Reputation: 61
Quote:
Originally Posted by rootaccess View Post
I never used 'X' in rsync, but I read it preserves extended attributes, not sure what that means yet.
One thing I have experienced is that if you don't use X you are not allowed to ping as user when you have restored the backup. I don't know if there are other extended attributes used.

Quote:
Originally Posted by rootaccess View Post
I just tried the setup you gave me and it did copy everything including hidden folders and directories. i was using 'rsync .*' from /. Not even sure what that did. Now I want to try restoring. I do like your method better. You copy proc and sys folders but not the contents. Smart. But either way, a newly installed system would have those files there. Is that where things get messed up? Should they be emptied out completely after doing a livecd rsync restore?
proc and sys should be emptied at shutdown.

I have never restored to a running system. If you restore from a Live CD then rsync will delete everything in proc and sys, if you use --delete. I forgot to mention that. Usually when I restore a backup I format the partition first. The only time I didn't, I messed it up because I forgot to use the delete-flag...
 
Old 11-17-2012, 12:56 PM   #8
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
Makes sense on the --delete flag, but I see you totally format the partition first so that isn't necessary unless you didn't format.

I was trying to make a minimal or regular install first, and then copy the files over. I also didn't do it from a livecd either. Your method makes alot more sense, I mean, why would someone make a minimal install, and then restore an entire system when everything is already backed up?

Finally, what do you do when there are multiple partitions involved or even LVMs? Seems like a nightmare if trying to restore to another system with the same LVM setup or no LVM setup for the new machine. I guess you have to format the new machine exactly as the source, down to the cylinder. I guess that's why theres kickstart for new installs, then maybe do the rsync on a livecd using the --delete flag?

Last edited by rootaccess; 11-17-2012 at 01:13 PM.
 
Old 11-17-2012, 01:18 PM   #9
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
I guess a good question is, what is going on in a production environment? Since i am studying to become a RHCE. Do they use rsync and restore full systems from backup? I would think not, especially with how you mention little things go wrong like not being able to ping and who knows what else. If you make a kickstart install to 100 machines, how do you tweak the systems all to reflect all the security and the modified files for each client like LDAP or anything, ssh, dhcp, etc?

I guess an rsync restore doesn't seem like it would pan well in an enterprise environment but I don't know, I'm curious whats really going on out there.
 
Old 11-17-2012, 01:26 PM   #10
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 302

Rep: Reputation: 61
I haven't used LVM so I know nothing about it.

If you want the restored system to have separate partitions for /home or /boot, just create the mountpoints and mount them before running rsync.

If the root partition is mounted at /mnt:
Code:
mkdir /mnt/home
mount /dev/sdXY /mnt/home
 
Old 11-17-2012, 01:36 PM   #11
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
Yes that I understand already. I guess I wanted to know more about backing up from LVM to LVM or "migrating".

If you did do a kickstart install and you just wanted to transfer the data and config files over but not the packages and libraries (which really seem to mess things up at least for me, would copying /etc be ok or is there alot more to it than just /etc?
 
Old 11-17-2012, 01:54 PM   #12
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 302

Rep: Reputation: 61
Arch has a wikipage about fullsystem backup with rsync There it says one should use -aAXv. Man rsync: "-A, --acls preserve ACLs (implies -p)" I don't know what that is...

I don't know about RHCE.

I use rsync to "install" (mostly) Xubuntu on a lot of machines, that anyway should be very much the same. Those machines are used as clients for surfing the web and whatever. So no servers on them. So the things I always change is fstab and grub to make them boot, and then hostname.
 
Old 11-17-2012, 02:19 PM   #13
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Lennie View Post
Arch has a wikipage about fullsystem backup with rsync There it says one should use -aAXv. Man rsync: "-A, --acls preserve ACLs (implies -p)" I don't know what that is...

I don't know about RHCE.

I use rsync to "install" (mostly) Xubuntu on a lot of machines, that anyway should be very much the same. Those machines are used as clients for surfing the web and whatever. So no servers on them. So the things I always change is fstab and grub to make them boot, and then hostname.
ACLs are Access Control Lists. I just studied that right now. Say a group called 'physics' own a directory called /physics and chmod everything 660 and the dir 770 so only physics can get in. But lets say they want to grant some user special priviledges to view a file or two in that directory but they aren't from the 'physics' group. They (or maybe just root himself) can grant access to that user to view a specific file in their directory and thats it with ACLs.

Yes what you are describing is for home use and works well. I do it too. Actually I just use clonezilla and it works like a charm. Never had an issue with it. only things that need changing are networking (host files, hostname, etc) and fstab and grub like you said. It doesn't copy proc and sys folders

So I guess rsync preserves these ACLs. Tar doesn't.

Last edited by rootaccess; 11-17-2012 at 02:21 PM.
 
Old 11-17-2012, 02:26 PM   #14
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 302

Rep: Reputation: 61
One thing I have thought about is the logs. It doesn't look like a freshly installed system if the logs are from several months back... (But it doesn't matter on those machines I maintain.)

I guess normally backup (with rsync or any other tool) is not used as a way to install a new system, but to restore the same system after a crash...
 
Old 11-17-2012, 02:42 PM   #15
rootaccess
Member
 
Registered: Mar 2012
Posts: 211

Original Poster
Rep: Reputation: Disabled
I was thinking the same thing. It can't be used in enterprise with too many variables going wrong. I think the only things necessary are the actual data files. Rsync is good for doing incremental backups. If you wanted to find only the files that have been changed in the last 24 hours in /home only and prune out the .mozilla directory and rsync them over SSH, here is the syntax..

Code:
find /home -wholename /home/*/.mozilla -prune -o -mtime -1 -print0 | rsync -0 -av --files-from=- / -e ssh root@192.168.0.110:/backups/`date '+%d-%B-%Y'`
So that would be good for small daily backups of the actual data but I would think other than data and configuration maintained in /etc, the rest of the system means nothing. It can always be re-installed.

Im new to enterprise but I'm guessing they probably do a bunch of kickstarts and install the necessary packages they need: LDAP, apache (if it isn't by default, can't remember) and post scripts. I would try to find a way to edit all the conf files in /etc on a massive scale so I wouldn't have to sit at 1000 machines doing that. Unless copying /etc to all of them would do just fine. I would assume
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
restoring an rsync backup Cultist Linux - Software 1 07-10-2011 09:18 AM
rsync backup system baronsam Linux - Software 9 05-22-2010 11:57 AM
Question about restoring a backup in a different system tirengarfio Linux - Newbie 7 01-21-2010 05:54 PM
Restoring from rsync backup a_l_a_n Linux - Software 3 09-12-2007 02:13 PM
Restoring from an rsync backup of / eldoran Linux - General 3 04-26-2006 12:34 AM


All times are GMT -5. The time now is 11:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration