LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-15-2015, 07:12 PM   #1
jzoudavy
Member
 
Registered: Apr 2012
Distribution: Ubuntu, SUSE, Redhat
Posts: 180

Rep: Reputation: Disabled
rsync questions


Hi all

I got two questions about rsync:

1. I am copying a huge dir with lots of files, how do I see the overall progress? I put in the progress flag but now it show the progress each individual files.

2. Is rsync slower than cp? I used CP to copy over a NFS mount and that took about an hour. I am using rsync to copy over the same mount over the same NFS and so far it has taken 10 minutes to copy 7G. I got over 150G of stuff.

I just used rsync -avh --progress ...

Thanks
Davy
 
Old 04-15-2015, 09:16 PM   #2
ShadowCat8
Member
 
Registered: Nov 2004
Location: Arcadia, CA
Distribution: Gentoo, Arch, (RedHat4.x-9.x, FedoraCore 1.x-4.x, Debian Potato-Sarge, LFS 6.0, etc.)
Posts: 220

Rep: Reputation: 46
Greetings,

The way we do it at the office to follow the general progress of rsync is as follows:
Code:
 ~ # rsync -av /source/directory/ /destination/directory/ &> ~/todaysDate-sync.log &
 ~ # tail -f todaysDate-sync.log
And, now you are tailing the output of what is being copied. If you want to see the filesystem size-change progress, you can do a:
Code:
 ~ # watch -n2 "du -sh /destination/directory/"
in another terminal to watch the size changes (every two seconds in the above example.)

And, yes, rsync is slower than cp, but has a LOT more functionality and safety built in. Generally, if you are starting with a blank destination, it's better to use cp, but if you are trying to get two filesystems in sync, it's better to use rsync. But, removing the "--progress" from your command should give you an immediate performance increase on the rsync copy times.

HTH. Let us know.

Last edited by ShadowCat8; 04-15-2015 at 09:20 PM. Reason: Added helpful suggestion to OPs command.
 
Old 04-15-2015, 09:17 PM   #3
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,333
Blog Entries: 12

Rep: Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729
rsync has some features which make it suited to repetitive backups in a way in which cp is not. One of those features is that, with the right switches, it does not overwrite an existing file in the target directory if 1)that file exists and 2) is unchanged in the target directory. In other words, in subsequent usages, it backs up only new or altered files.

Consequently, the first time rsync is used, it may take quite a while, but subsequent usages with the same source and target can be much, much quicker.

I'm by no means an rsync expert--I figured out the formulation that worked for me (rsync -rav) and have used it since. The Arch wiki, though, is an excellent reference.
 
Old 04-15-2015, 09:43 PM   #4
bishopolis
LQ Newbie
 
Registered: Feb 2015
Posts: 5

Rep: Reputation: Disabled
Quote:
Originally Posted by jzoudavy View Post
Hi all

I [have] two questions about rsync:

1. I am copying a huge dir with lots of files[;] how do I see the overall progress? I put in the progress flag but now it [shows] the progress [of] each individual [file].
That's normal. There is no way to tell how much data, nor how much time an rsync will take. Given that, what would you like a progress bar to say?

Quote:
Originally Posted by jzoudavy View Post
2. Is rsync slower than cp? I used CP to copy over [an] NFS mount, and that took about an hour. I am using rsync to copy over the same mount over the same NFS and so far it has taken 10 minutes to copy 7G. I [have] over 150G of stuff.
So, we should expect 200 minutes, by your observations so far. That's if it's linear.

Yes, Rsync is slower at being cp than cp. CP, though, is not very good at being a network-efficient re-startable synchronizer for data, so use the best tool for the job.

You will find rsync needs a lot of time at the start of a synchronization run as it calculates the work. CP doesn't care what's there already, and will appear faster for less work. Copy files over an unreliable link, though, and the benefit in rsync will become obvious quickly.
 
Old 04-15-2015, 10:02 PM   #5
jzoudavy
Member
 
Registered: Apr 2012
Distribution: Ubuntu, SUSE, Redhat
Posts: 180

Original Poster
Rep: Reputation: Disabled
hi frankbell,

do you know what switch allows for not overwriting? is it the -u?



Quote:
Originally Posted by frankbell View Post
rsync has some features which make it suited to repetitive backups in a way in which cp is not. One of those features is that, with the right switches, it does not overwrite an existing file in the target directory if 1)that file exists and 2) is unchanged in the target directory. In other words, in subsequent usages, it backs up only new or altered files.

Consequently, the first time rsync is used, it may take quite a while, but subsequent usages with the same source and target can be much, much quicker.

I'm by no means an rsync expert--I figured out the formulation that worked for me (rsync -rav) and have used it since. The Arch wiki, though, is an excellent reference.
 
Old 04-15-2015, 10:33 PM   #6
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,333
Blog Entries: 12

Rep: Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729
It's been a while since I did my research, but, if I recall correctly, it's "-a" for "archive." "-r" means "recursive" (include subdirectories) and "-v" means "verbose."

From the man page:

Code:
-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
If I recall incorrectly, I'm sure someone will correct me.

As an aside, rsync is a very versatile command, which means it and its permutations can also be quite complex. A web search for "rsync examples" can turn up many useful links that will be much more helpful than the man page.

Last edited by frankbell; 04-15-2015 at 10:38 PM.
 
Old 04-16-2015, 12:16 AM   #7
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,831
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Please note that -r and -a are redundant because -a includes -r. You can see that in the documentation frankbell referenced where -a basically means -rlptgoD. It doesn't hurt to include it. I'm only mentioning it's not necessary when using -a.

Code:
rsync -av src dst
Is my go-to command as well, like frankbell mentioned. That mode of operation already only copies over changes without copying it over if it exists and the checksum is the same.
 
1 members found this post helpful.
Old 04-16-2015, 07:08 PM   #8
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,333
Blog Entries: 12

Rep: Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729Reputation: 2729
Thanks, sag47.

When I started looking into rsync, I found a collection of the most confusing tech webpages I have ever seen. I finally found something that worked and stuck with it. I'll try refining it in the light of your advice next time I back up.

The man page wasn't much help, because it lacks examples; lack of examples is the Achilles heel of the man format.
 
Old 04-16-2015, 07:20 PM   #9
Lnthink
Member
 
Registered: May 2010
Location: Lafayette, LA
Distribution: Ubuntu, RH, Fedora
Posts: 38

Rep: Reputation: 8
cp also has the nasty problem of messing with timestamps, messing with ownerships, etc.
It also has the nasty habit of not handling symbolic links for files cleanly, and will turn a symbolic link into a full file copy.

Rsync handles all these things beautifully.

About the only think that I can think of that it doesn't handle well is device files, the only thing that I know of that handles this correctly is cpio.
Unknown to me is how well rsync handles hard links. I've not tested this, you could end up with multiple files instead of a file by inode, and multiple file names to that inode, as per normal. I've not tested/observed this closely.
 
Old 04-16-2015, 10:27 PM   #10
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,831
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Quote:
Originally Posted by Lnthink View Post
About the only think that I can think of that it doesn't handle well is device files, the only thing that I know of that handles this correctly is cpio.
Unknown to me is how well rsync handles hard links. I've not tested this, you could end up with multiple files instead of a file by inode, and multiple file names to that inode, as per normal. I've not tested/observed this closely.
In what way does it not handle device files for you well? Perhaps, you're missing an option you could add to rsync. Regarding hardlinks.

Quote:
Originally Posted by rsync man page
Code:
Note that -a does not preserve hardlinks, because
    finding multiply-linked files is expensive. You must separately specify -H. 
...
-H, --hard-links
    This tells rsync to look for hard-linked files in the transfer and link together the corresponding files on the receiving side. Without this option, hard-linked files in the transfer are treated as though they were separate files. 
When you are updating a non-empty destination, this option only ensures
    that files that are hard-linked together on the source are hard-linked together on the destination. It does NOT currently endeavor to break already existing hard links on the destination that do not exist between the source files. Note, however, that if one or more extra-linked files have content changes, they will become unlinked when updated (assuming you are not using the --inplace option). 
Note that rsync can only detect hard links between files that are inside
    the transfer set. If rsync updates a file that has extra hard-link connections to files outside the transfer, that linkage will be broken. If you are tempted to use the --inplace option to avoid this breakage, be very careful that you know how your files are being updated so that you are certain that no unintended changes happen due to lingering hard links (and see the --inplace option for more caveats). 
If incremental recursion is active (see --recursive), rsync may transfer
    a missing hard-linked file before it finds that another link for that contents exists elsewhere in the hierarchy. This does not affect the accuracy of the transfer, just its efficiency. One way to avoid this is to disable incremental recursion using the --no-inc-recursive option. 
...
Also, regarding the -a option (aka -rlptgoD). You can read up each option individually to learn what -a does e.g. -r -l -p -t -g -o -D. There are options for handling special devices and the like.
 
Old 04-17-2015, 04:04 PM   #11
ShadowCat8
Member
 
Registered: Nov 2004
Location: Arcadia, CA
Distribution: Gentoo, Arch, (RedHat4.x-9.x, FedoraCore 1.x-4.x, Debian Potato-Sarge, LFS 6.0, etc.)
Posts: 220

Rep: Reputation: 46
Quote:
Originally Posted by jzoudavy View Post
hi frankbell,

do you know what switch allows for not overwriting? is it the -u?
Greetings,

Yes, what you need is to let rsync know what to do with the files you are about to replace with --backup and --suffix=<SUFFIX>
Quote:
Originally Posted by rsync man page
...

-b, --backup - make backups (see --suffix & --backup-dir)
--backup-dir=DIR - make backups into hierarchy based in DIR
--suffix=SUFFIX - backup suffix (default ~ w/o --backup-dir)

...
So, if you want to get 'dest' in sync to 'src' and back up all the files that will be replaced with the today's date as a suffix, you would do something like this:
Code:
 ~ $ rsync -av --backup --suffix=.20150417 /src/ /dest/
Hope that helps. Let us know.
 
1 members found this post helpful.
Old 04-18-2015, 07:34 AM   #12
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,831
Blog Entries: 36

Rep: Reputation: 451Reputation: 451Reputation: 451Reputation: 451Reputation: 451
Quote:
Originally Posted by jzoudavy View Post
2. Is rsync slower than cp? I used CP to copy over a NFS mount and that took about an hour. I am using rsync to copy over the same mount over the same NFS and so far it has taken 10 minutes to copy 7G. I got over 150G of stuff.

I just used rsync -avh --progress ...

Thanks
Davy
Hi Davy,
When you say copying the "same amount over," do you mean the *exact* same files? If it's not the exact same files then the time can't be used as a good benchmark between the two utilities (cp and rsync) because transfer times vary depending on file size and amount of files.

Also, using rsync copies files at 12MB/s (7GB in 10 minutes) vs 42MB/s using cp (150GB in 60 minutes) according to your post. It's likely the difference is due to the number of files and the average file size in the dataset you're comparing.

In general, I prefer rsync because it guarantees the integrity of the copied file.

Last edited by sag47; 04-18-2015 at 07:37 AM.
 
Old 04-18-2015, 09:57 AM   #13
JeremyBoden
Member
 
Registered: Nov 2011
Posts: 937

Rep: Reputation: 174Reputation: 174
If you copy these files periodically:-

copy will always take one hour to copy the 150GB, but rsync will only copy the modifications (new, changed, deleted etc).

Obviously, rsync will see 150GB of changes on the first run - but future runs will be much faster than a copy.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Some more Rsync questions sigint-ninja Linux - Newbie 15 11-22-2013 09:54 AM
rsync questions jsteel Linux - General 2 07-05-2010 10:52 AM
rsync questions crazy8 Linux - Newbie 11 04-05-2008 01:59 PM
rsync questions slack66 Linux - General 4 11-05-2005 01:31 AM
rsync questions csross Linux - General 3 06-25-2004 11:10 AM


All times are GMT -5. The time now is 06:01 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration