[SOLVED] rsync: Can't turn off recursion

andrew.comly · 11-01-2015, 10:16 PM

I would like to sync the folder ~/.maxima and all it's *.ini files to another folder, without copying/syncing its' folders/subdirectories. Unfortunatedly when I use rsync it not only updates the *.ini files, but also copies over ~/.maxima's subfolder: binary.

My preamble:

Code:

#!/bin/bash

#VARIABLES
#Initialize
unset BKROOT HMA

#Define
BKROOT=/media/username/Seagate/comp1
HMA=~

#wxMaxima
	if [ -d ${BKROOT}${HMA}/.maxima ]; then
		ATTEMPTS 1-4
	else
		cp -apr --parents ~/.maxima/*.ini ${BKROOT}/
	fi

ATTEMPTS 1-4
So far I've tried:

Code:

1) cp -apr --parents ~/.maxima/*.ini ${BKROOT}
2) rsync -Savi -Eu --no-r -stat --delete ~/.maxima ${BKROOT}${HMA}
3) rsync -Savi -Eu -stat --delete ~/.maxima/*.ini ${BKROOT}${HMA}/.maxima
4) rsync -Savi -Eu --exclude 'binary' -stat --delete ~/.maxima ${BKROOT}${HMA}

RESULTS
1) The cp command works great for the first backup when the directory ${BKROOT} doesn't exist yet, but in case you delete a file you don't need any more in the source folder, using this the next backup won't delete that file from the target directory.

2) Gets rid of old files you don't want, but it ALWAYS copies over the undesirable subfolder "binary".

3) This is nice because it won't copy over anything other than a *.ini file (even folders/directories it doesn't copy), but when you delete an old *.ini file on the source, it won't delete that file on the target.

4) A strength here is that it doesn't copy over that specific directory, but it does still copy over other folders/directories and other files (e.g. text.log, a.txt etc.)

Can rsync just sync "*.ini" files while excluding subdirectories/subfolders? Strange how in the rsync man pages there is no "maxdepth=" option.

berndbausch · 11-01-2015, 11:24 PM

Attempt 2 probably wants --stats instead of -stat. If you type -stat with a single dash (and without the final s), rsync interprets this as -s -t -a, and the -a reverses the earlier --no-r.

Now if you try

Code:

rsync -Savi -Eu --no-r --stats --delete ~/.maxima ${BKROOT}${HMA}

you will get the error message that --delete requires --dirs or --recursive. Add --dirs and all should be fine. Perhaps

berndbausch · 11-01-2015, 11:36 PM

Quote:

Originally Posted by berndbausch

you will get the error message that --delete requires --dirs or --recursive. Add --dirs and all should be fine

Not quite, actually. This will not create .maxima on the target, and it will also copy the binary directory, though without content.

So I guess it requires a little more work, but perhaps it's a good start.

andrew.comly · 12-12-2015, 07:37 AM

Quote:

Originally Posted by berndbausch

Attempt 2 probably wants --stats instead of -stat. If you type -stat with a single dash (and without the final s), rsync interprets this as -s -t -a, and the -a reverses the earlier --no-r.

Actually this just uncovers the fact that I never actually realized that in order to get a verbose set of statistics on the file transfer, one shouldn't use "-stat" nor even "-stats"; The option one should use is "--stats"! I really had been screwing this up for quite some time. Blame it on LibreOffice autocorrect, which always autochanges "--" to a long "-". ;(...Better late than never!)

Quote:

Originally Posted by berndbausch

Now if you try

Code:

rsync -Savi -Eu --no-r --stats --delete ~/.maxima ${BKROOT}${HMA}

you will get the error message that --delete requires --dirs or --recursive. Add --dirs and all should be fine. Perhaps

I attempted:

Code:

REMOTE=media/a/LG/local;  MYHOME=home/a
$ rsync -Sai -Eu --stats --no-r --delete --dirs /home/a/.maxima /${REMOTE}/${MYHOME}
cd+++++++++ .maxima/

Number of files: 1 (dir: 1)
Number of created files: 1 (dir: 1)
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 63
Total bytes received: 26

sent 63 bytes  received 26 bytes  178.00 bytes/sec
total size is 0  speedup is 0.00

$ find /home/a/.maxima    #displays everything in source folder
/home/a/.maxima
/home/a/.maxima/pm2.ini
/home/a/.maxima/am.ini
/home/a/.maxima/pm.ini
/home/a/.maxima/binary
/home/a/.maxima/binary/binary-gcl
/home/a/.maxima/binary/binary-gcl/share
/home/a/.maxima/binary/binary-gcl/share/draw
/home/a/.maxima/binary/binary-gcl/share/draw/gnuplot.o
/home/a/.maxima/binary/binary-gcl/share/draw/picture.data
/home/a/.maxima/binary/binary-gcl/share/draw/grcommon.o
/home/a/.maxima/binary/binary-gcl/share/draw/vtk.o
/home/a/.maxima/binary/binary-gcl/share/draw/grcommon.data
/home/a/.maxima/binary/binary-gcl/share/draw/vtk.data
/home/a/.maxima/binary/binary-gcl/share/draw/picture.o
/home/a/.maxima/binary/binary-gcl/share/draw/gnuplot.data
/home/a/.maxima/New.txt
Destination:
$ find local/home/a/.maxima/             #displays everything in destination folder
local/home/a/.maxima/

Thus it looks like "--no-r" takes "no recursive" one more step than I participated: it JUST copies over the directory, ITSELF. This really shows how computers take things quite literally.

andrew.comly · 12-12-2015, 08:14 AM

ORIGINAL PROBLEM: I would like to sync the folder ~/.maxima and all it's *.ini files to another folder, without copying/syncing its' folders/subdirectories.

Recently I have learned two new rsync options: 1) --exclude/include; 2) --prune-empty-dirs.

One can complete the above job with --exclude/include:

Method01

Code:

rsync -Sai -Eu --stats --exclude binary --exclude *.txt /${MYHOME}/.maxima /${REMOTE}/${MYHOME}
cd+++++++++ .maxima/
>f+++++++++ .maxima/am.ini
>f+++++++++ .maxima/pm.ini
>f+++++++++ .maxima/pm2.ini

Number of files: 4 (reg: 3, dir: 1)

For this small situation method01 works. But what about a more versatile solution for a more complex directory structure containing more types of files? For this we need --prune-empty-dirs:

Method02:

Code:

$ rsync -Sai -Eu --stats --include *.ini --prune-empty-dirs -f "+ */" -f "- *" /${MYHOME}/.maxima /${REMOTE}/${MYHOME}
cd+++++++++ .maxima/
>f+++++++++ .maxima/am.ini
>f+++++++++ .maxima/pm.ini
>f+++++++++ .maxima/pm2.ini

Number of files: 9 (reg: 3, dir: 6)
Number of created files: 4 (reg: 3, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 3

#RSYNC manual file:
3155-3157) The combination of "+ */", "+ *.c", and "- *" would include all
directories and C source files but nothing else (see also the
--prune-empty-dirs option)
1801-1803) -F option is a shorthand for adding two --filter rules to your command.

This worked quite well. The '+' in '-f "+ */"' means include and the '*/' means directories(recursive). In -f "- *", the "-" means exclude and the "*" means any files.

But can't --prune itself include *.ini files also?

Method03:

Code:

$ rsync -Sai -Eu --stats --prune-empty-dirs -f "+ */" -f "+ *.ini" -f "- *" /${MYHOME}/.maxima /${REMOTE}/${MYHOME}

This is as effective as method02, but more concise.

But what if I had made some unwanted files on the destination(e.g. text.txt)? rsync's --prune-empty-dirs is also compatible with "--delete"! And remember, we don't need the "--no-r" option, thus no need to slavishly annex "--dirs" after "--delete":

METHOD04:

Code:

$ rsync -Sai -Eu --stats --delete --prune-empty-dirs -f "+ */" -f "+ *.ini" -f "- *" /${MYHOME}/.maxima /${REMOTE}/${MYHOME}
*deleting   .maxima/text.txt/
.d..t...... .maxima/

My personal favorite is METHOD04. Hope some of you can better understand rsync's --prune-empty-dirs versatility!