LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-22-2019, 01:46 PM   #1
MALDATA
Member
 
Registered: Mar 2005
Posts: 157

Rep: Reputation: 19
Can't get rsync exclude to work correctly


Hi all,

This has been bothering me for a while, and I can't figure out where the problem is. I use ssh & rsync to do remote backups. On the machine I'm backing up, there is a "VirtualBox VMs" directory that is huge and doesn't need to be backed up. I also don't need to back up all the hidden directories in home. So, I run

Code:
rsync -avSXe "ssh -p <port>" --delete --exclude=".*/" --exclude="VirtualBox\ VMs/" host:/home/user/ /home/user/backup
This correctly excludes all hidden directories, but it still rsyncs the VM directory. I've tried all different kinds of syntax, for example,

Code:
rsync -avSXe "ssh -p <port>" --delete --exclude=".*/" --exclude={"VirtualBox\ VMs/"} host:/home/user/ /home/user/backup
Code:
rsync -avSXe "ssh -p <port>" --delete --exclude={".*/","VirtualBox\ VMs/"} host:/home/user/ /home/user/backup
Code:
rsync -avSXe "ssh -p <port>" --delete --exclude=".*/" --exclude={"VirtualBox\ VMs/"} host:/home/user/ /home/user/backup
Code:
rsync -avSXe "ssh -p <port>" --delete --exclude=".*/" --exclude="VirtualBox*/" host:/home/user/ /home/user/backup
And so on. None of them have worked. It rsyncs the VM directory every time. What am I missing? Thanks!
 
Old 10-22-2019, 02:19 PM   #2
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Code:
--exclude=PATTERN       exclude files matching PATTERN
the man rsync is a little confusing with its explanation of PATTERN

a section of it ( you should read the whole section, but it is confusing )
Quote:
if the pattern contains a / (not counting a trailing /) or a "**", then it is matched against the full pathname, including any
leading directories. If the pattern doesn’t contain a / or a "**", then it is matched only against the final component of the
filename. (Remember that the algorithm is applied recursively so "full filename" can actually be any portion of a path from the
starting directory on down.)

a trailing "dir_name/***" will match both the directory (as if "dir_name/" had been specified) and everything in the directory
(as if "dir_name/**" had been specified). This behaviour was added in version 2.6.7.

I would have to experiment with dir_name/*** and dir_name/** is see what the difference is, I'm not sure I understand what is in the manual fully

Last edited by Firerat; 10-22-2019 at 02:21 PM.
 
Old 10-22-2019, 02:38 PM   #3
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.9.2009
Posts: 5,708

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
I believe the exclude path needs to be absolute, not relative:
Code:
--exclude="/home/user/VirtualBox\ VMs/"
...and I'd sure take that space out of the directory name...but that's just me
 
Old 10-22-2019, 02:45 PM   #4
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
some more from man rsync

Quote:
o if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched
against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus "/foo" would match a name of
"foo" at either the "root of the transfer" (for a global rule) or in the merge-file’s directory (for a per-directory rule). An
unqualified "foo" would match a name of "foo" anywhere in the tree because the algorithm is applied recursively from the top
down; it behaves as if each path component gets a turn at being the end of the filename. Even the unanchored "sub/foo" would
match at any point in the hierarchy where a "foo" was found within a directory named "sub". See the section on ANCHORING IN‐
CLUDE/EXCLUDE PATTERNS for a full discussion of how to specify a pattern that matches at the root of the transfer.

o if the pattern ends with a / then it will only match a directory, not a regular file, symlink, or device.

o rsync chooses between doing a simple string match and wildcard matching by checking if the pattern contains one of these three
wildcard characters: ’*’, ’?’, and ’[’ .

o a ’*’ matches any path component, but it stops at slashes.

o use ’**’ to match anything, including slashes.

o a ’?’ matches any character except a slash (/).

o a ’[’ introduces a character class, such as [a-z] or [[:alpha:]].


so something like

Code:
--exclude="VirtualBox\ VMs/**"
is probably enough

if not
Code:
--exclude="/**VirtualBox\ VMs/**"
 
1 members found this post helpful.
Old 10-22-2019, 03:16 PM   #5
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.9.2009
Posts: 5,708

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
^ You're probably right. As you say, that part of the man page is pretty confusing and not at all clear.

I wonder of that the OP is using is not working because rsync isn't handling the escaped space as expected. As I read the man page, their
Code:
--exclude="VirtualBox\ VMs/"
should work.

Perhaps the \ isn't needed, given the double quotes?

Or maybe try
Code:
--exclude="VirtualBox?VMs/"
where the ? is the single character wild card?
 
1 members found this post helpful.
Old 10-22-2019, 05:36 PM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,770

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
Quote:
Originally Posted by scasey View Post
I wonder of that the OP is using is not working because rsync isn't handling the escaped space as expected. As I read the man page, their
Code:
--exclude="VirtualBox\ VMs/"
should work.

Perhaps the \ isn't needed, given the double quotes?
Indeed. For the shell, inside double quotes the backslash character retains its special meaning only when followed by one of five specific characters: $, `, ", \, or <newline>. The sequence "\ " is passed by the shell as a literal backslash character followed by a space.

Last edited by rknichols; 10-22-2019 at 05:37 PM.
 
2 members found this post helpful.
Old 10-22-2019, 10:35 PM   #7
MALDATA
Member
 
Registered: Mar 2005
Posts: 157

Original Poster
Rep: Reputation: 19
Quote:
Indeed. For the shell, inside double quotes the backslash character retains its special meaning only when followed by one of five specific characters: $, `, ", \, or <newline>. The sequence "\ " is passed by the shell as a literal backslash character followed by a space.
If I interpret this correctly, I should try without the backslash to escape the space:
Code:
--exclude="VirtualBox VMs/"
I'm relatively certain I tried that at one point, but I'll give it another shot tomorrow. I'm surprised this hasn't come up more often. Thanks for the help!
 
Old 10-22-2019, 10:40 PM   #8
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.9.2009
Posts: 5,708

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
Quote:
Originally Posted by MALDATA View Post
If I interpret this correctly, I should try without the backslash to escape the space:
Code:
--exclude="VirtualBox VMs/"
I'm relatively certain I tried that at one point, but I'll give it another shot tomorrow. I'm surprised this hasn't come up more often. Thanks for the help!
Please let us know whether or not that worked...

[nag]Again: Consider removing the space from the directory name entirely. OK. Won’t mention it again [\nag]
 
Old 10-23-2019, 09:24 AM   #9
MALDATA
Member
 
Registered: Mar 2005
Posts: 157

Original Poster
Rep: Reputation: 19
[SOLVED] Can't get rsync exclude to work correctly

This works as expected, with no escape character before the space:
Code:
--exclude={".*/","VirtualBox VMs/"}
I thought I tried that at one point, but I guess not. Truth be told, after using bash regularly 15 years, I still can't remember the rules for single and double quotes, escape characters, etc.

Quote:
Again: Consider removing the space from the directory name entirely.
100% agree. It's already gone. But I needed to figure out why this wasn't working, for the next time when I don't have the option to change the directory name. And for the next person out there that runs into this problem!

Thanks everyone!
 
2 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync exclude arguments held in variable not passing correctly (bourne script) Archy1 Programming 4 07-28-2016 10:17 AM
[SOLVED] rsync --exclude-from not working correctly? talkinggoat Linux - Newbie 3 04-19-2016 09:44 PM
Can we use exclude option in"rm" command to exclude some files/folders? yadav_rk727 Linux - Newbie 1 02-03-2010 10:14 AM
can't get rsync exclude files Batta Linux - Newbie 1 03-27-2006 10:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration