LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 06-23-2016, 11:22 AM   #1
LouArnold
Member
 
Registered: Mar 2006
Distribution: Linux Mint 18.1 w/Cinnamon
Posts: 75

Rep: Reputation: 15
Help: Rsync options/switches to copy directory tree - unique files only?


I run a home CentOS 7 system. I want to modify some of the OS, but want to copy a directory tree (one large one) to a data drive. Links seem to be a problem. Some trials have show circular copying - via a link the same subdirectories and files are copied again and again. I don't mind the links being copied (and path modified for the data drive), but files should only be copied once.

What is the proper command and switches/options?
 
Old 06-23-2016, 01:10 PM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 6,082
Blog Entries: 3

Rep: Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170
You don't mention the exact options you are using to try to run "rsync" If you use the option -a, it is the equivalent of -rlptgoD and thus copies the symlinks also because of the -l. If you list the options individually, you can exclude the -l with -rptgoDv as your options. That will then sync all the files and directories that are not symlinks. However, that will ignore symlinks completely.

If you need to copy ACLs or hardlinks then add -A and -H to your formula.

Last edited by Turbocapitalist; 06-23-2016 at 01:11 PM.
 
Old 06-23-2016, 03:59 PM   #3
LouArnold
Member
 
Registered: Mar 2006
Distribution: Linux Mint 18.1 w/Cinnamon
Posts: 75

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Turbocapitalist View Post
You don't mention the exact options you are using to try to run "rsync" If you use the option -a, it is the equivalent of -rlptgoD and thus copies the symlinks also because of the -l. If you list the options individually, you can exclude the -l with -rptgoDv as your options. That will then sync all the files and directories that are not symlinks. However, that will ignore symlinks completely.

If you need to copy ACLs or hardlinks then add -A and -H to your formula.
I understand about links, but what controls the cyclical copying? If rsync finds links in the source that point to files already copied, it will copy them again unless some switch/option stops it. I want to avoid the copying.

The other side of this is the link to the data drive itself. I need to tell rsync to omit this path "/mnt/WD500". I don't know how to do that.

Last edited by LouArnold; 06-23-2016 at 04:51 PM.
 
Old 06-24-2016, 12:33 AM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 6,082
Blog Entries: 3

Rep: Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170
By design, "rsync" only copies the changes it finds. So if a file is unchanged, it is not copied a second time. However, if a file is changed, only the changes themselves are copied.

You can use the --exclude option to omit a path or some files. See the manual page for "rsync" in the section entitled "INCLUDE/EXCLUDE PATTERN RULES" for the specifications on how to form the right pattern for your needs. That section is worth reading as it explains quite a lot.
 
Old 06-24-2016, 01:34 PM   #5
LouArnold
Member
 
Registered: Mar 2006
Distribution: Linux Mint 18.1 w/Cinnamon
Posts: 75

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Turbocapitalist View Post
By design, "rsync" only copies the changes it finds. So if a file is unchanged, it is not copied a second time. However, if a file is changed, only the changes themselves are copied.
What you wrote applies to two different sessions, does it not? When the same files are attempted as part of one command, will it also avoid copying a second and third time, to the same point in a directory tree, during that command?


Quote:
You can use the --exclude option to omit a path or some files. See the manual page for "rsync" in the section entitled "INCLUDE/EXCLUDE PATTERN RULES" for the specifications on how to form the right pattern for your needs. That section is worth reading as it explains quite a lot.
I went and read that section again, but its still not clear. If I want to omit the tree from WD500 and below will "--exclude=/mnt/WD500" do the trick? (WD500 is the mount point for the data drive).
 
Old 06-24-2016, 02:52 PM   #6
LouArnold
Member
 
Registered: Mar 2006
Distribution: Linux Mint 18.1 w/Cinnamon
Posts: 75

Original Poster
Rep: Reputation: 15
I tried the command (below) and I think it worked.
# rsync -av --exclude=/mnt/WD500 / /mnt/WD500/ShoeSambaX/01-RootBkup
It excluded the directory WD500 and the subdirectories and file. That's what I wanted. But I also got a number of the "file has vanished" lines with samples below.

Code:
file has vanished: "/proc/10/task/10/exe"
file has vanished: "/proc/101/exe"
file has vanished: "/proc/101/task/101/exe"
file has vanished: "/proc/11/exe"
This apparently normal.

Also got this in summary:
Code:
sent 14,670,647 bytes  received 1,546,223 bytes  1,544,463.81 bytes/sec
total size is 140,751,613,830,845  speedup is 8,679,332.93 (DRY RUN)
Can't quite figure that one out.

Directories backed up:
boot, dev, etc, home (empty). media (empty), mnt (all excluded), opt, proc, root, run, tmp, var. There were 20 folders under "/".
 
Old 06-24-2016, 03:08 PM   #7
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 6,082
Blog Entries: 3

Rep: Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170Reputation: 3170
You'll want to exclude dev, media, mnt, proc, run, and tmp.

Code:
man 7 hier
man 5 proc
man mknod
 
Old 06-24-2016, 03:43 PM   #8
BCarey
Senior Member
 
Registered: Oct 2005
Location: New Mexico
Distribution: Slackware
Posts: 1,639

Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
You'll want to exclude dev, media, mnt, proc, run, and tmp.

Code:
man 7 hier
man 5 proc
man mknod
...and to add to that, rsync has an --exclude-from option which allows you to create a file which contains the directories you want to exclude so you don't have to add them to your command line (and remember them).
 
Old 06-24-2016, 04:56 PM   #9
LouArnold
Member
 
Registered: Mar 2006
Distribution: Linux Mint 18.1 w/Cinnamon
Posts: 75

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by BCarey View Post
...and to add to that, rsync has an --exclude-from option which allows you to create a file which contains the directories you want to exclude so you don't have to add them to your command line (and remember them).
I tried the --exclude-from file - that part works. But a trial with an actual copying produced a file over 250 GB, so its copying things from the data drive or copying files via different links.

I'll check the man pages, thanks.
The rsync command I used was:
Code:
rsync -av --exclude-from=/mnt/WD500/ShoeSambaX/rsyncnotes/excludelist.txt / /mnt/WD500/ShoeSambaX/01-RootBkup >/rsynctest4.txt
The excludelist file:
Code:
# excludelist.txt for rsync backup from root ("/").
# there are normally 20 folders backed up, but some folders are not needed.
/mnt/WD500
/proc
/sys
/var/lib
/var/log
/var/spool
/var/tmp
/var/webmin
/var/named
/var/cache
/tmp
/run
/root
/usr/lib
/usr/lib64
/usr/src
/usr/share
/usr/bin
/usr/include
/usr/java
/usr/libexec
/usr/games
/usr/local
/usr/sbin
 
  


Reply

Tags
copying, rsync, tree


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
RSYNC Help - Copy Only Changed Files to New Directory euphoricrhino Linux - Software 1 11-23-2011 04:29 PM
command tofind files by extension through a directory tree, and copy them all to the budword Linux - Newbie 2 06-27-2009 01:48 PM
How to copy files from an entire directory tree using terminal commands??? lusius188 Linux - General 3 04-10-2006 12:23 PM
locate and copy files - without directory tree rodda Linux - General 1 01-16-2005 03:10 AM
How to copy a directory tree without copying the files in it? system Linux - General 2 01-15-2002 10:36 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration