rsync --delete question

jms89 · 02-10-2013, 01:56 PM

Hi, I am using rsycn with the --delete command, but I want to prevent certain directories from being removed with this command, that are not directly part of the --exclude-from list.

Here's a schematic of what my two directory structures look like:

source disk directory: /foo /excluded_directories
target disk directory: /foo /remote_stuff

Here "/foo" respresents a group of directories which I am interested in syncing and deleting files from, and /remote_stuff are directories on the target disk which are not on the source disk, and which I want left alone.

If I just apply rsycn like --exclude-from "exlude list" --delete /source/ /target/ then the /remote_stuff directories will be deleted. I don't want this, how can I prevent it?

Thanks!

Skaperen · 02-10-2013, 02:20 PM

It should not delete anything that is in the exclude list (as long as --delete-excluded is not used). Can you put these directories in the exclude list? If you want those directories to be "add enw files but do not delete existing files that do not exist in the source" then I think you need to do two runs of rsync. One run would focus (source and destination) on the subdirectory to be updated without delete, and one run on the whole tree with that subdirectory excluded.

marozsas · 02-10-2013, 04:02 PM

You have to build a small bash program around your rsync.
You have not to rsync /foo, but instead /foo/allowed_dir_1 /foo/allowed_dir_2, etc...
I did this with the help of the following code (it is not the entire code, it is just the section that matter for this discussion).
As you can see, the exclude list is a extend grep pattern format. I list the content of current source dir and exclude from this listing the folders I don't want to sync. Then, I run rsync for each allowed dir inside the for loop.

Code:

RSYNC="/usr/bin/rsync"
src="/home"
dst="homeusr@${host}::home"
flags="-az -Ap --delete"
exclude="lost+found|PrivateArea|Softwares|former-employees"
export RSYNC_PASSWORD="mySecretPassword"

cd ${src}

for dir in $(\ls -d | grep -Ev "${exclude}"); do
	if [ "$verbose" == "1" ]; then
		echo "running: nice -n 19 $RSYNC $flags --exclude "${exclude}" $dir $dst"
	fi
	nice -n 19 $RSYNC $flags $dir $dst
done

cheers,

jms89 · 02-10-2013, 08:46 PM

Quote:

Originally Posted by marozsas

You have to build a small bash program around your rsync.
You have not to rsync /foo, but instead /foo/allowed_dir_1 /foo/allowed_dir_2, etc...
I did this with the help of the following code (it is not the entire code, it is just the section that matter for this discussion).
As you can see, the exclude list is a extend grep pattern format. I list the content of current source dir and exclude from this listing the folders I don't want to sync. Then, I run rsync for each allowed dir inside the for loop.

Code:

RSYNC="/usr/bin/rsync"
src="/home"
dst="homeusr@${host}::home"
flags="-az -Ap --delete"
exclude="lost+found|PrivateArea|Softwares|former-employees"
export RSYNC_PASSWORD="mySecretPassword"

cd ${src}

for dir in $(\ls -d | grep -Ev "${exclude}"); do
	if [ "$verbose" == "1" ]; then
		echo "running: nice -n 19 $RSYNC $flags --exclude "${exclude}" $dir $dst"
	fi
	nice -n 19 $RSYNC $flags $dir $dst
done

cheers,

Cool, this makes sense. I'm a little new to bash scripting so I don't understand all of the syntax, but I do understand what you're doing. Just curious, why the nice command though? I don't see why this is should be neccessary, does it speed things up for you?

marozsas · 02-11-2013, 06:14 AM

No, its is just the opposite. The nice makes the things slower, with a low priority.
The reason for this is this sync is done in a production machine, every 15min or so, and I don't want to impact other process with the sync.
Of course, if this is not an issue, you can safely omit the nice command so the line becomes just "$RSYNC $flags $dir $dst". After the shell substitutions it becomes something like this: "/usr/bin/rsync -az -Ap --delete allowed_dir_n homeusr@remotemachinename::home".

cheers,