LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-11-2020, 05:56 PM   #1
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Rep: Reputation: 28
rsync --exclude-from command line option not working on macOS using bash shell


macOS 10.15.5 on an iMac

I'm trying to run the rsync command to back up my primary disk to an external hard drive on the same system (no network involved).

I can get it to work as expected except for the "--exclude-from=FILE" command line option. I'm wondering if I'm doing something wrong or if this command line option is not implemented (or disabled) on macOS.

The command I'm playing with is this:
Code:
$ cd
$ rsync -R -n -v -r --files-from="/Users/rhimbo/.rsync-files-from.txt" --exclude-from="/Users/rhimbo/.rsync-exclude-from.txt" --exclude="/Downloads/Digital Editions/" . /Volumes/WD-disk2s1 > file-list.txt
The "~/Downloads/Digital Editions" directory tree is excluded from the back up as expected. However, if I specify this same directory tree using the "--exclude-from=FILE" command line option and specified FILE (.rsync-exclude-from.txt) it seems to be ignored.

The contents of the .rsync-exclude-from.txt file is:
Code:
exclude, /Downloads/Digital Editions/
I've tried all of these combinations:
/Downloads/Digital Editions
/Downloads/Digital Editions/
Downloads/Digital Editions
Downloads/Digital Editions/
Digital Editions
Digital Editions/

I've been pouring over the two man page sections INCLUDE/EXCLUDE PATTERN RULES and ANCHORING INCLUDE/EXCLUDE PATTERNS but I cannot see what I'm doing wrong.
 
Old 06-11-2020, 06:56 PM   #2
HappyTux
Senior Member
 
Registered: Mar 2003
Location: Nova Scotia, Canada
Distribution: Debian AMD64
Posts: 3,672

Rep: Reputation: 119Reputation: 119
Quote:
Originally Posted by rhimbo View Post
macOS 10.15.5 on an iMac


The contents of the .rsync-exclude-from.txt file is:
Code:
exclude, /Downloads/Digital Editions/
but I cannot see what I'm doing wrong.
Unless you have a directory /Downloads/Digital or Editions/ in the current directory the script is run in then it will not find them due to the unescaped space. It will treat them as separate items in the bash shell. Try.

Code:
exclude, /Downloads/Digital\ Editions/
Edit: And if the file had been included I would have answered the question on the Macrumors forum but I dislike having to ask questions to answer one.

Last edited by HappyTux; 06-11-2020 at 07:08 PM.
 
Old 06-11-2020, 08:54 PM   #3
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
[QUOTE=HappyTux;6133302]Unless you have a directory /Downloads/Digital or Editions/ in the current directory the script is run in then it will not find them due to the unescaped space. It will treat them as separate items in the bash shell. Try.

Code:
exclude, /Downloads/Digital\ Editions/
Code:
$ cd
$ ls -F
ARISExpress/		Downloads/		Public/			foo/
Applications/		Library/		Sites/			rsync-errors.txt
Books/			Movies/			bar/			run-rsync.sh
Desktop/		Music/			dev/			scratch/
Documents/		Pictures/		file-list.txt
$ 
$ 
$ cd Downloads
$ $ ls -lF
total 429384
-rw-r--r--@  1 rhimbo  staff     229183 Jan 17 20:25 1509.0066v1.pdf
-rw-r--r--@  1 rhimbo  staff     989535 Jan 17 20:22 1907.0424v2.pdf
-rw-r--r--@  1 rhimbo  staff    1273741 Jan 17 20:23 1912.0332v3.pdf
-rw-r--r--@  1 rhimbo  staff    1131934 Jan 17 20:23 2001.0188v1.pdf
-rw-r--r--@  1 rhimbo  staff    1401730 Jul  8  2017 Cybersecurity-for-Things-Part-1.pdf
drwxrwxrwx   6 rhimbo  staff        192 Jun  7  2013 Digital Editions/
-rw-r--r--@  1 rhimbo  staff     125059 Jan 17 20:31 ECE_and_Spacetime.pdf
-rw-r--r--@  1 rhimbo  staff   66578155 Jun  7 14:18 Emacs-26.3-universal.dmg
-rw-r--r--@  1 rhimbo  staff      62269 Oct 22  2017 Intel-Core-i5-4200U-Processor-3M-Cache-up-to-2.60-GHz-Product-Specifications.pdf
I should have included this in my original post. I tried all of the combinations below in addition to the ones listed in the original post; none of them worked.
Code:
exclude, Digital\ Editions
exclude, Digital?Editions 
exclude, "Digital\ Editions"
exclude, "Digital?Editions"
Nevertheless, I tried it anyway and it did nothing, as if I never specified the "--files-from" command line option at all. Curiously, The man page does not say that it's necessary to quote or escape any part of the pattern. For example, "*" in a pattern is not interpreted by the shell. At least it isn't from my results experimenting.

Additionally, the man page clearly states that there can be only one pattern per line in a file containing include/exclude arguments. If that is true, then it would make sense that one would not need to escape anything to the right of the filter rule. Not being one to trust the accuracy of documentation, I tried it in every combination that I thought would be a way to help understand the behavior.

But, alas, still befuddled....

Last edited by rhimbo; 06-11-2020 at 09:53 PM. Reason: Added contents of "Downloads" directory to show that there is a "Digital Editions" directory.
 
Old 06-12-2020, 12:20 AM   #4
HappyTux
Senior Member
 
Registered: Mar 2003
Location: Nova Scotia, Canada
Distribution: Debian AMD64
Posts: 3,672

Rep: Reputation: 119Reputation: 119
Quote:
Originally Posted by rhimbo View Post

But, alas, still befuddled....
I am not, took a few minutes but I got it.

Code:
MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l 
total 8
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 Space 2/
-rw-r--r--@ 1 MacUser2525  staff    13B 12 Jun 02:02 exclude.txt
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 folder 1/
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 z 3/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ rsync -avz --exclude-from 'exclude.txt'  /Volumes/Sea_To_Do/Downloads/test /Volumes/Files/
sending incremental file list
test/
test/Space 2/
test/folder 1/
test/z 3/

sent 146 bytes  received 32 bytes  356.00 bytes/sec
total size is 0  speedup is 0.00

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ cat exclude.txt 
exclude.txt

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l /Volumes/Files/test/
total 0
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 Space 2/
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 folder 1/
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 z 3/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ rm -r /Volumes/Files/test/*

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l /Volumes/Files/test/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ nano exclude.txt


MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ cat exclude.txt 
test/folder*

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ rsync -avz --exclude-from 'exclude.txt'  /Volumes/Sea_To_Do/Downloads/test /Volumes/Files/
sending incremental file list
test/exclude.txt

sent 194 bytes  received 38 bytes  464.00 bytes/sec
total size is 13  speedup is 0.06



MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l /Volumes/Files/test/
total 8
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 Space 2/
-rw-r--r--  1 MacUser2525  staff    13B 12 Jun 02:02 exclude.txt
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 folder 1/
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 z 3/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ nano exclude.txt 

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ rm -r /Volumes/Files/test/*

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l /Volumes/Files/test/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ rsync -avz --exclude-from 'exclude.txt'  /Volumes/Sea_To_Do/Downloads/test /Volumes/Files/
sending incremental file list
test/
test/exclude.txt
test/Space 2/
test/z 3/

sent 202 bytes  received 51 bytes  506.00 bytes/sec
total size is 8  speedup is 0.03

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ ls -l /Volumes/Files/test/
total 8
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 Space 2/
-rw-r--r--  1 MacUser2525  staff     8B 12 Jun 02:04 exclude.txt
drwxr-xr-x  2 MacUser2525  staff    68B 12 Jun 01:29 z 3/

MacUser2525:/Volumes/Sea_To_Do/Downloads/test$ cat exclude.txt 
folder*
All thanks to this wonderful page.

https://www.thegeekstuff.com/2011/01...s-and-folders/

Done on OSX Mojave using rsync 3.1.2 I have installed, not the >10 year old mac version.

Edit: And if needing a more modern version of it then try the dmg you can get here linked on this page.

https://github.com/rsyncOSX/RsyncOSX

Last edited by HappyTux; 06-12-2020 at 12:34 AM.
 
Old 06-12-2020, 12:52 AM   #5
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,056

Rep: Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469
Please try to find a way to avoid spaces in filenames. Also, when in doubt, use absolute path-names:
Code:
exclude, /Users/rhimbo/Downloads/DigitalEditions
 
Old 06-12-2020, 09:24 AM   #6
HappyTux
Senior Member
 
Registered: Mar 2003
Location: Nova Scotia, Canada
Distribution: Debian AMD64
Posts: 3,672

Rep: Reputation: 119Reputation: 119
Quote:
Originally Posted by NevemTeve View Post
Please try to find a way to avoid spaces in filenames. Also, when in doubt, use absolute path-names:
Code:
exclude, /Users/rhimbo/Downloads/DigitalEditions
I am certain I read in the linked page that the paths are all relative when using it. I agree with the no spaces but Apple in their wisdom sets up every computer by default with a Macintosh HD startup volume name. Training their user to do the same. I always have spaces removed in my file names for easy access in the terminal for processing.
 
Old 06-12-2020, 02:22 PM   #7
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
Quote:
Originally Posted by HappyTux View Post
I am certain I read in the linked page that the paths are all relative when using it. I agree with the no spaces but Apple in their wisdom sets up every computer by default with a Macintosh HD startup volume name. Training their user to do the same. I always have spaces removed in my file names for easy access in the terminal for processing.
Well, the destination paths are relative to the SRC if the -R or --relative command line options are specified. So, if I have a /Users/rhimbo/Downloads directory, then the SRC files will be written to /<dest volume>/Users/rhimbo/Downloads/... if I specify -R or --relative. However, if I do not specify -R or --relative, the files will be written to /<dest volume>/Downloads/....

However, I will try your suggestion of removing spaces. I'm also going to get a more modern rsync as you suggested. At this point I think either the rsync I have is broken or the man page does not coincide with the version I currently have installed.

Code:
$ rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
 
Old 06-12-2020, 02:23 PM   #8
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
Quote:
Originally Posted by NevemTeve View Post
Please try to find a way to avoid spaces in filenames. Also, when in doubt, use absolute path-names:
Code:
exclude, /Users/rhimbo/Downloads/DigitalEditions
I'll try to rename the SRC directories to remove all spaces. Thanks...
 
Old 06-12-2020, 05:04 PM   #9
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
By the way, as I said in my original post, I can get it to work as expected when putting the filter specifications on the command line using the "--exclude" command line option. I cannot get it to work using the "--exclude-from"=FILE command line option, which specifies that the filter patterns to exclude "files" (files and directories) be read from the specified FILE.

Curiously, rsync does not complain if I type "--exclude FILE" or "--exclude=FILE" contrary to the man page. Hmm....
 
Old 06-12-2020, 05:52 PM   #10
HappyTux
Senior Member
 
Registered: Mar 2003
Location: Nova Scotia, Canada
Distribution: Debian AMD64
Posts: 3,672

Rep: Reputation: 119Reputation: 119
Quote:
Originally Posted by rhimbo View Post
By the way, as I said in my original post, I can get it to work as expected when putting the filter specifications on the command line using the "--exclude" command line option. I cannot get it to work using the "--exclude-from"=FILE command line option, which specifies that the filter patterns to exclude "files" (files and directories) be read from the specified FILE.

Curiously, rsync does not complain if I type "--exclude FILE" or "--exclude=FILE" contrary to the man page. Hmm....
I tried it with the = there for the " --exclude-from 'FILE.txt' " and it would not work with it included, only without = did it work as expected.
 
Old 06-12-2020, 07:37 PM   #11
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
OK here is the latest. Apparently, at least using rsync version 2.6.9 on macOS Catalina, the man page is just totally wrong in so many ways. But I'll just summarize here.

Apparently, the "filter patterns" in the file you pass to the "--exclude-from" command line argument do not follow the syntax of the man page. That is, they are not filter patterns as documented in the man page. You should think of these entries as strings which rsync will pass to the shell directly.

Directories seemingly are not recognized. For example, the following were ignored.

Downloads/Digital?Editions
Documents/work/src-jdk/
Documents/Zoom


However, these entries were processed.

Digital?Editions
src-jdk/
Zoom


A leading "/" is interpreted as the "root" directory, not the "root of the transfer" as the man page says. Not that the man page defines that term, but one can infer that it means the directory represented as SRC in the command SYNOPSIS of the man page. Other discrepancies abound such as wild card characters.

Thanks to everyone for enthusiastically lending support.

Last edited by rhimbo; 06-12-2020 at 07:44 PM.
 
Old 06-12-2020, 07:45 PM   #12
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
I'm going to mark this as "solved" but I will happily read additional posts....
 
Old 06-13-2020, 12:48 AM   #13
rhimbo
Member
 
Registered: Dec 2004
Distribution: Ubuntu 19.04 on Lenova ThinkPad T440
Posts: 132

Original Poster
Rep: Reputation: 28
Well, I spoke too soon. Permissions are not preserved on the transferred files at the destination. I tried "-a" option but also the explicit "-l -p -t -g -o -D" which "-a" supposedly represents. More problems with the Apple version.

At least I have a backup now. I'll figure out the permission problem when I upgrade to a newer rsync version. So this is just an FYI note for y'all.....
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Using Find with an exclude/exclude file metallica1973 Linux - General 8 11-06-2011 09:39 PM
Can we use exclude option in"rm" command to exclude some files/folders? yadav_rk727 Linux - Newbie 1 02-03-2010 10:14 AM
CVS Exclude : Exclude sub directories from check out On Linux from command line shajay12 Linux - Newbie 1 08-03-2009 12:36 AM
Using the exclude option absolutely with rsync karlovac Programming 0 03-01-2004 12:39 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 05:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration