LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-01-2009, 02:41 PM   #1
jlinkels
LQ Guru
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,191

Rep: Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039
using rsync for copying multiple partitions


Regularly I find myself cloning a machine using rsync. I find it understandable, reliable and fast, faster than dd, and I don't have to worry about different partition sizes etc.

However, usually I partition my hard disk in a number of partitions:
Code:
/
/home
/usr
/var
When I start with a new, empty machine, I start up with a USB stick or live CD, and my new, empty hard disk becomes /dev/sdb. After creating the 4 partitions I have /dev/sdb1, /dev/sdb2... etc. My root directory is on the disk I used for booting, usually /dev/sda.

So, in order to access my newly created partitions, I mount them on the /mnt/directory of my root:
Code:
mounted now    later
/mnt/sdb1      /
/mnt/sdb2      /home
/mnt/sdb3      /usr
/mnt/sdb4      /var
In other words, I mount now /dev/sdb1 on /mnt/sdb1, while after copying /dev/sdb1 will become my root directory, /dev/sdb2 become my /home directory, etc.

When I start the resync process to copy the image from a remote machine, I have to copy all 4 partitions separately. First the root directory, excluding /home, /usr, /var, then /home, then /usr, /var, like this:
Code:
action 1:
rsync --exclude='/home' --exclude='/var' --exclude='/usr' my.remote.machine:/ /dev/sdb1/

action 2:
rsync my.remote.machine:/home /dev/sdb2/

action 3:
rsync my.remote.machine:/usr /dev/sdb3/

action 4:
rsync my.remote.machine:/var /dev/sdb4/
That is a lot of typing and waiting. Sometimes I have a different partition scheme so it is not really feasible to write a script to use always.

Now the QUESTION: is there a smarter way of mounting the newly formatted disk (/dev/sdb1, /dev/sdb2... etc) in my root tree so I can perform the rsync copy in just one time, without all the excludes, but assuring that the correct source partitions end up on the correct destination partitions?

jlinkels
 
Old 09-01-2009, 03:15 PM   #2
ShadowCat8
Member
 
Registered: Nov 2004
Location: San Bernardino, CA
Distribution: Gentoo, Arch, (RedHat4.x-9.x, FedoraCore 1.x-4.x, Debian Potato-Sarge, LFS 6.0, etc.)
Posts: 258

Rep: Reputation: 52
Well, I would recommend doing the same thing that they do regularly with Gentoo from their LiveCDs; e.g. mount the new partitions in the structure they will be used in, and then do a batch transfer with rsync of the overall filesystem, excluding the /proc and /dev of the running system you are using as a source. (Actually, with the Gentoo LiveCDs, they use tarballs after the basic structure has been created; /boot and /usr for their base created by the user. The tarball creates the rest of the filesystem by just unpacking it at /.)

In other words:
Code:
# mount -t ext3 /dev/sdb1 /mnt/newdrive
# cd /mnt/newdrive/
# mkdir home usr var proc dev
# mount -t ext3 /dev/sdb2 /mnt/newdrive/home ; mount -t ext3 /dev/sdb3 /mnt/newdrive/usr ; mount -t ext3 /dev/sdb4 /mnt/newdrive/var
# rsync --exclude='/proc' --exclude='/dev' the.remote.machine:/ /mnt/newdrive/
Now, of course, since you are in that directory at that time, you can use './' instead of '/mnt/newdrive' for the mounting, but I was being cautious.

HTH.
 
Old 09-01-2009, 03:25 PM   #3
jlinkels
LQ Guru
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,191

Original Poster
Rep: Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039Reputation: 1039
I see what you do, and I remember I tried it and it went wrong, that is everything was copied into my /mnt/newdrive partition.

But what you say sounds so logical, once /dev/sdb2 is mounted on /mnt/newdrive/home, anything coming from remotehost:/home should be copied in there, shouldn't it?

The next few days I have a machine coming in which has to be cloned, I'll try and report back.

I omitted the exclusion of /proc and /dev in my example. But I think I should skip /proc and /sys, and copy /dev otherwise I don't have the correct nodes for my devices, right?

Thanks

jlinkels
 
Old 10-06-2009, 06:33 PM   #4
ShadowCat8
Member
 
Registered: Nov 2004
Location: San Bernardino, CA
Distribution: Gentoo, Arch, (RedHat4.x-9.x, FedoraCore 1.x-4.x, Debian Potato-Sarge, LFS 6.0, etc.)
Posts: 258

Rep: Reputation: 52
Well, if you are using udev, you don't want to copy anything into /dev. Anything the system will need will be created in /dev/ by udev.

Otherwise, you are correct. Copy /dev/ if you are not using udev.

HTH.
 
Old 10-06-2009, 06:54 PM   #5
lutusp
Member
 
Registered: Sep 2009
Distribution: Fedora
Posts: 835

Rep: Reputation: 102Reputation: 102
Quote:
Originally Posted by jlinkels View Post
Regularly I find myself cloning a machine using rsync. I find it understandable, reliable and fast, faster than dd, and I don't have to worry about different partition sizes etc.

However, usually I partition my hard disk in a number of partitions:
Code:
/
/home
/usr
/var
Okay, just stop. Don't partition your HDD like this. Don't you realize the number of full-partition failure modes is equal to the number of partitions, and that you cannot possibly guess which one will fill first?

Rather than try to solve the problem you have created for yourself, collapse all the system partitions into one. That will create much more reliable system, and it will solve the problem you are trying to solve right now.

Three partitions: system (/) including /home, /usr, /var, /tmp etc., then swap, then everything else. Very simple. The system can still fail because of a full partition, but it won't fail while there's plenty of space but on the wrong partitions, which is the problem with many small partitions.
 
Old 10-06-2009, 07:04 PM   #6
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,824

Rep: Reputation: 615Reputation: 615Reputation: 615Reputation: 615Reputation: 615Reputation: 615
There's a case for /home and maybe /opt (if used) on separate partitions. Good to save if you reinstall the OS.
 
Old 10-06-2009, 07:09 PM   #7
lutusp
Member
 
Registered: Sep 2009
Distribution: Fedora
Posts: 835

Rep: Reputation: 102Reputation: 102
Quote:
Originally Posted by AlucardZero View Post
There's a case for /home and maybe /opt (if used) on separate partitions. Good to save if you reinstall the OS.
True, but this fashion with many partitions - one each for /usr, var, /tmp, etc. -- is crazy. The risk is that one of the system partitions will fill up while there's free space available elsewhere.

I prefer having precisely one disk-full failure mode: when the disk is actually full.
 
Old 10-07-2009, 05:57 PM   #8
manwithaplan
Member
 
Registered: Nov 2008
Location: ~/
Distribution: Arch || Sidux
Posts: 393

Rep: Reputation: 45
I have a script that can do just this ... You'll need to add the mounts & add more directories in the variables set... You'll also have to set the destination variable... should be an easy edit.

The script runs rolling updates at different times I set in a cron. And only adds the changes from the previous timestamp.

I'll post the script if you want to give it a try.
 
Old 11-02-2009, 08:09 PM   #9
ShadowCat8
Member
 
Registered: Nov 2004
Location: San Bernardino, CA
Distribution: Gentoo, Arch, (RedHat4.x-9.x, FedoraCore 1.x-4.x, Debian Potato-Sarge, LFS 6.0, etc.)
Posts: 258

Rep: Reputation: 52
Quote:
Originally Posted by lutusp View Post
True, but this fashion with many partitions - one each for /usr, var, /tmp, etc. -- is crazy. The risk is that one of the system partitions will fill up while there's free space available elsewhere.

I prefer having precisely one disk-full failure mode: when the disk is actually full.
I can see 2 reasons why just one flat partition for your entire disk (save for the swap partition) is a bad idea, too, though.

1.) Disk checks/scans. Generally, when you have to do a virus/malware check, you should only really have to scan your home directory(-ies)... and maybe /tmp. They should be the only places that you as a regular user can write to. And, you shouldn't be doing anything but administration as root. Also, it will likely improve your system performance during the regular filesystem checks as the system boots since it seems common for the filesystem root and the other partitions to get out of sync as far as the fstab's dump sequence is concerned. (Remember: Mounting / as /mnt/gentoo in LiveCD counts as a mount.)

2.) Filesystem corruption from system power failure. One of the more common failures I've seen on an overall filesystem after a power failure is:
Code:
Wrong fs type, bad option, bad superblock on /dev/sda1
missing codepage or helper program, or other error
In some cases useful info is found in syslog  try
dmesg | tail  or so
I think we've all seen this error message more than enough. And, from this, if you are about to do your recovery from a LiveCD and do an e2fsck from there, you will likely have file fragments in lost+found from *many* different files across the filesystem. On the other hand, if you have /home on another partition, then it is likely that the only corruption you will see in that partition is any file that was actively being accessed at the time of the power failure. An example of that was the system that had Firefox open when their system died; The bookmarks.html file was corrupted and moved to lost+found, but nothing else came up as bad in /home.

@lutusp: I understand your point, too, of generally not knowing how large each of the other major directories are going to grow. The difference of space consumption in just /usr is going to vary greatly between just building or not building OpenOffice, let alone any number of other popular large programs getting installed. If *I* were to recommend a partitioning scheme to your average linux end-user, I think it would go something like this:
Code:
Partition          Mountpoint          General Size
=========          ==========          ============
First               /boot               ~50MB
Second              <swap>              (At least double installed RAM)
Third               /                   ~1/3rd of total remaining space on the drive
Fourth              /home               The remaining ~2/3rds space on the drive
Now, you can vary the ratio of / versus /home, but I find that as much as users like to install programs on their systems, they will generally fill up more space than their programs do. Also, for swap, a general rule I've followed is to create a swap that is twice as large as the installed RAM for a regular system and four times the installed RAM on a server that is running a database.

Another thing that I do with my drives is switch /home and the swap partitions in the layout above. The issue with putting the swap at the end is your average end-user may have problems doing their space calculations for the drive. In such cases, stick with the above layout. It is much simpler.

Just my 2.

HTH.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync: error while copying Fusi Linux - Software 5 06-10-2013 12:00 AM
Copying whole changed files - rsync? deadlock Linux - Software 2 12-04-2007 05:57 AM
Rsync not copying symlink davidb2002 Linux - Software 9 10-24-2007 04:17 PM
prevent rsync from copying open files? bkeeper Linux - General 2 12-11-2006 05:16 AM
Copying from Slackware 10 to Debian Sarge using rsync apolinsky Slackware 5 10-07-2004 10:17 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 12:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration