LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 06-28-2011, 02:49 PM   #1
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Rep: Reputation: 2
Faster rsync -aH for large directories?


Hello All

I have a system that has two hard drives. I use it to make backup's of my various computers. A cron job runs at 3am the following:

rsync -aH /mnt/hd1/ /mnt/hd2/

Note that the -H preserves hard links. Problem is I am sitting here at 1pm and it is still trying to finish that job!!! The transfer rate is around 100kB/s.

I do not get it because it is a simple hard drive to hard drive copy. It should rip through that at maximum throughput. What is going on? Looking at the top command CPU is maxed and memory is at 300% (ie using swap).

Is there a way to make rsync faster on this?

Thanks
Mike
 
Old 06-29-2011, 08:04 AM   #2
Skaperen
Senior Member
 
Registered: May 2009
Location: center of singularity
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,678
Blog Entries: 31

Rep: Reputation: 176Reputation: 176
Suppose that a previous run of rsync transferred some files, and this second run of rsync, for which more files are now to be transferred, involves filenames that are hardlinks of files that still exist but were previously transferred. Now also suppose that the new hardlink names are earlier in the sorted order rsync operates in. In order for rsync to detect that a new earlier name is hardlinked to an old later name, so that it doesn't end up just duplicating that file on the target, it has to map (read all the names and inode numbers) the entire tree you are copying.

So how many total filenames are involved?
 
Old 06-29-2011, 08:05 AM   #3
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
By default rsync is using ssh style encryption. Sometimes for large transfers between internal machines it is worthwhile to temporarily enable something like rsh then tell rsync to use that as the shell since it doesn't do encryption.
 
Old 06-29-2011, 12:19 PM   #4
Skaperen
Senior Member
 
Registered: May 2009
Location: center of singularity
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,678
Blog Entries: 31

Rep: Reputation: 176Reputation: 176
Another way around the slow ssh encryption that rsync could be using is to use my "rsend" script. To use rsend, open a shell on each of the source and target machines under the appropriate userid. On the source machine, run "rsend <path>" or just cd to the desired path and run "rsend". It picks a port number (or use the -p option to force a port number) and runs a temporary rsync daemon in the foreground to serve that path (a directory or just a file). It outputs some sample commands you could copy and paste into the target machine shell session to fetch the directory tree or file. Omit the "-z" option if your network is faster than your compression.

http://phil.ipal.org/scripts/rsend

Append ".txt" to the URL if your browser has trouble saving the file (then rename it once saved).
 
1 members found this post helpful.
Old 06-29-2011, 08:52 PM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
Actually, as this is just disk to disk on the same system, definitely skip the encryption option; its pointless.
 
Old 06-29-2011, 09:39 PM   #6
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hello All

First up I want to thank you all for your helpful advice.

I know about the encryption issue. It is not a problem here though it is all on the same machine. In that case rsync opens a simple pipe where no encryption is applied. You can verify that with the "lsof | grep rsync" command.

Thanks Phil for rsend, I will have to look through it, I am not familiar with rsync in daemon mode, I take it that it does not rely on ssh? this would be interesting for other transfers...

I have delved deeper into the problem. The rsync -aH backup is the last command after all the other workstations have copied their data to this one machine. It turns out that there were other problems with a previously interupted cp command. I am not sure to what extent this made rsync slow though. The offending script has been repaired and tested and the rsync -aH now set up in cron. So tomorrow I will know if rsync -aH is still a problem. Fingers crossed...

Mike
 
Old 06-29-2011, 11:54 PM   #7
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
Why not set cron to go off in 5mins time; just to check its ok?
 
Old 06-30-2011, 07:32 AM   #8
Skaperen
Senior Member
 
Registered: May 2009
Location: center of singularity
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,678
Blog Entries: 31

Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by pilotmm View Post
Thanks Phil for rsend, I will have to look through it, I am not familiar with rsync in daemon mode, I take it that it does not rely on ssh? this would be interesting for other transfers...
It uses the rsync protocol directly. Other tools like stunnel can put it through encryption. With ssh, you can, of course, always do rsync directly, but could also pipe the rsync protocol through port forwarding, too. I've done rsync via stunnel when I needed to spread one tree of files to other machines where my desktop wasn't involved in the data path, but the direct path between those machines still went over the wild internet.


Quote:
Originally Posted by pilotmm View Post
I have delved deeper into the problem. The rsync -aH backup is the last command after all the other workstations have copied their data to this one machine. It turns out that there were other problems with a previously interupted cp command. I am not sure to what extent this made rsync slow though. The offending script has been repaired and tested and the rsync -aH now set up in cron. So tomorrow I will know if rsync -aH is still a problem. Fingers crossed...

Mike
Also, rsync can be slower to start once both source and target have file trees, compared to when the target is empty, due to that full scan of each file tree to see what is to be copied and what is to be linked.
 
Old 06-30-2011, 07:48 AM   #9
Karl Godt
Member
 
Registered: Mar 2010
Location: Kiel , Germany
Distribution: once:SuSE6.2,Debian3.1, aurox9.2+3,Mandrake?,DSL? then:W7st,WVHB, #!8.10.02,PUPPY4.3.1 now:Macpup
Posts: 314

Rep: Reputation: 45
The
Quote:
-a, --archive archive mode; same as -rlptgoD (no -H)
option includes several single options , which I haven't used all by now ,

but as far as I can tell , the
Quote:
-k, --copy-dirlinks transform symlink to a dir into referent dir
option for example into /usr takes forever .

I seem to stay with
Code:
-urlR --devices --progress
the last option is nice if there are not too many files to update '-u' in showing the status of the progress .
 
Old 06-30-2011, 04:03 PM   #10
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hi Karl Godt

Thanks for the option break-down... -H is absolutely necessary... there are about 40+ rsnapshot's dereferencing all that would surely crash my system. I suspect that is where the stall is coming from. Sorting out the full extent of what is xfered and what is link on large directories might be the problem.... however this problem has evolved. It is upstream processes that I am struggling with for now.... The interrupted cp command is no longer happening but the system is choking on another upstream process.

Thanks for your help. This is a slow process, all these backup scripts seem to be taking 10+ hours a piece. So every tweak takes a full day to confirm whether it makes a difference. Once the script is running I do not want to interrupt it because I don't want backups corrupted. I know I can do better but just haven't found the bottle neck. The current dragon is discussed on the link below... problem is this is all on hacked hardware...

http://forum.dsmg600.info/viewtopic.php?id=6661

Once I get these other problems solved I am will be back here trying to solve the original -aH problem....

Thanks
 
Old 07-01-2011, 07:39 AM   #11
Skaperen
Senior Member
 
Registered: May 2009
Location: center of singularity
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,678
Blog Entries: 31

Rep: Reputation: 176Reputation: 176
If the entire tree of what you are trying to send with rsync is incremental backups done by making a hard link copy of a subtree, such as daily or weekly, where you would keep older versions, then perhaps it might be better to trim that down by eliminating the older hard links.

If you also use rsync to populate that source tree with the hard links, consider the --backup-dir= option in rsync. What that will do is move deleted or replaced files into the specified directory tree, perhaps using the date as the name of that directory. These backup trees will not be replicas of the whole original as of that date, but instead will just be what was deleted or replaced on the next date.

If you need to recover a file as of a given date, start at that date and look for the file. If it is not there, move forward until it is found. If you need to recreate the exact tree in a way that it does not have files that didn't exist then, start with the more recent tree and copy it to the restore staging area, and repeat on each date until reaching the desired date, and then delete files which were not there based on keeping a file listing of each date made of that day's rsync run(s).
 
Old 07-02-2011, 02:31 AM   #12
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Ok I have some exciting news!!! I managed to get my main back-up script to complete its work within 10 minutes. This is much better than the 24+ hours!!! I still have to test the rsync -aH script.... Skaperen I think you might be right in trimming down how many incrementals I keep. Maybe I am too paranoid... Like I said it was upsteam processes I was battling. I think I slayed that dragon now!!! Tomorrow will be rsync -aH Hope it works.

Mike
 
Old 07-02-2011, 12:53 PM   #13
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hello All

Just writing to report that I managed to complete the rsync -aH script in 2 hours. This is reasonable. It seems that it was the upstream processes that were causing the delays. Thanks a lot to all of you for all the help you offered. Now unfortunately I didn't learn a special trick to make rsync -aH faster, but learned a lot about making effective backups in general.

Thanks to everyone that contributed to this thread. Albeit the original problem was merely a symptom of larger problems, much of your advice contributed to fixing these larger problems.

Let me pass on my wisdom, hopefully it will help others. First I need to summarize what I am trying to achieve. I have multiple machines in multiple locations and the goal was to build a set of scripts that will automatically backup all machines on a daily basis with incremental backups with a high level of redundancy. In summary every physical location has at least two hard drives that mirror all the data of all machines. Needless to say it is a complicated set-up and I learned a lot.

Let me list the problems/solutions
==================================

1) Some errors were causing cp commands to be interrupted leaving large volumes of work for rsync to clean up the mess

Lesson here is test incrementally instead of writing the whole process completely and letting it rip. Now I already know about this lesson, but every once in a while I convince myself that I am a good enough sys-admin/programmer that I can write it all at once. So maybe the real lesson is stay humble, your impatient ego is your worst enemy.

2) Many of the scripts were running too slow, some of them started to execute simultaneously causing transfer commands to read and write to directories that were in constant flux. Naturally this forced machines to be overloaded and slower causing even more scripts to execute. The worst thing is that the file structure becomes totally corrupted making each transfer slower overall.

Lesson here is time each script under worst case scenarios so first you understand roughly how long it is really going to take. Second when you have multiple scripts executed on multiple machines where sequential operation is absolutely mandatory ensure there is some logic to coordinate it. There are a couple approaches, one can use a set of lock files and -x tests to act as semaphores to force scripts to wait until another completes. And/or have a central machine use ssh to initiate each script in turn. I used a combination of this because I want the process to work in the off chance that communication between remote locations is lost (power outages etc.). So semaphores between remote xfers and central control on LAN xfers.

3) Many of the rsync commands were not behaving as expected. I wanted many of the scripts to ignore all those hidden files that programs like firefox are constantly creating. I put in things like --exclude '/.*' to filter those out. However the way I was passing it to rsync in the script was causing rsync to mis-interpret it and still transfer these files. Since these directories are changing constantly... this was creating much more incremental information to be created and deleted with every xfer. Thus generating a lot of unnecessary work.

Lesson here is to test each script with full verbosity to verify exactly what it is doing. Furthermore just to make sure all the options are correct create a set of test cases to fine tune each critical command before writing them into the actual script.

4) Originally I was lazy and I would use the transfer protocol that required the least amount of set-up for me. However it turned out that there are various limitations that caused many of the transfers to stall compounding a lot of the problems that I had.

Lesson here is again test and measure all the transfer protocols first to get a sense of the best way to move the data. Let me list what I learned here:
a) I found that transfers over an ssh pipe is the slowest. However it is also the most reliable and safest so for remote transfers it is the best.
b) Internal LAN transfers over NFS work well when reading off the nfs server or writing <50GB of data. Larger than that NFS chokes and will crawl at 50kB/s speeds.
c) For large internal transfers I recommend setting up an rsync daemon (try rsend). The rsync transfer protocol is not the most secure but you avoid the encryption overhead associated with ssh.
d) Finally this may seem obvious but transfers that are completely local to one machine (ie cp /from/ /to/), use ssh to initiate the transfer so the whole process runs locally on that machine.
 
Old 07-03-2011, 08:54 PM   #14
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
A couple of comments:

re 4c) rsync does have an option to use ssh; your choice as to whether you do or not

re 4d) how you connect to the system eg ssh has nothing to do with how the actual cmd works if its local cmd like eg local cp

That being said, that's a very nice write up
 
Old 07-04-2011, 03:53 PM   #15
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by chrism01 View Post
A couple of comments:

re 4c) rsync does have an option to use ssh; your choice as to whether you do or not
Is this an option in daemon mode? I am aware of ssh being an option in simple stand-alone mode.

In daemon mode I was trying to get around the extra overhead associated with encryption. If I do rsync with ssh I get xfer speeds of about 2MB/s. In simple Daemon mode the xfer is in plain text but the speed is 12-20 MB/s. Looking at the CPU the ssh xfer is CPU bound, while the rsync daemon mode transfer is bound by the LAN and/or hardware. This may not be a problem for fast server machines with extra clock cycles to give out... but most money for the main backup machine was put into hard drives and RAID controllers and little on CPU and RAM.

Quote:
Originally Posted by chrism01 View Post
re 4d) how you connect to the system eg ssh has nothing to do with how the actual cmd works if its local cmd like eg local cp
Sorry I should give a clarification here. Originally I was lazy and executing the cp command over NFS mounts. So the CPU of machine A would be handling the logic of cp while the actual data transfer would be between disks on machine B. I am not sure how NFS is implemented but there is a chance that all or a fraction of the xfer data is moved B -> A then A -> B. Overall this is very inefficient yet easy to write into a script. Allowing machine A to invoke the cp command to run locally on machine B completely dramatically increased the speed in this step. I used ssh on machine A to invoke a call to cp on machine B.

Quote:
Originally Posted by chrism01 View Post
That being said, that's a very nice write up
Thanks!!! I have learned a lot from forums and have received much help always happy to give back.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync can't handle large files??? deweirdt Linux - Server 19 02-28-2012 02:35 PM
rsync to check two directories...? resetreset Linux - Software 3 01-26-2011 03:53 AM
[SOLVED] large tar nightly + rsync to compare question sir-lancealot Linux - Server 5 01-05-2011 02:28 PM
[SOLVED] rsync -z vs. scp ... which one is faster over network? someshpr Linux - General 5 12-08-2010 06:06 PM
sorting through large file directories n_hendrick Linux - General 4 05-08-2007 12:08 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration