[SOLVED] rsync: cannot delete non-empty directory

andrew.comly · 01-12-2017, 11:50 PM

I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync.

PURPOSE:
1) Copy over all files/directories containing files ending in ".rc" FROM sourceHDD/.config TO destination/HDD/.config; (*.rc is of type file, not of type directory)
2) Delete any directory not recursively containing a file "*.rc" (not only empty directories).
3) Delete any file not containing a file "*.rc".

SITUATION:
I have a directory /home/${USER}/workshop/sourceHDD/.config, its file directory structure:

Code:

cd /home/${USER}/workshop
mkdir -p ./sourceHDD/.config/{dir1/dir2,dir3,dir5}
touch ./sourceHDD/.config/{file{1,2}.{txt,rc},dir3/file4.{rc,txt},dir1/dir2/file3.{rc,txt},dir5/new.txt,blah.txt}

I have a second directory /home/${USER}/workshop/destinationHDD/.config of the following file directory structure:

Code:

mkdir -p ./destinationHDD/.config/{blubber,surplus/waste/fat}
touch ./destinationHDD/.config/{blubber/trash1.txt,surplus/waste/fat/trash2.txt}

_________________________________________
ATTEMPT 01:
cd /home/${USER}/workshop
COMMAND:

Code:

rsync -avPpm --include='*rc*' --include='*/' --exclude='*' --delete ./source/* ./destination

RESULT:

Code:

9 files to consider
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus
cannot delete non-empty directory: .config/blubber
.config/
.config/file1.rc
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=7/9)
.config/file2.rc
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=6/9)
.config/dir1/
.config/dir1/dir2/
.config/dir1/dir2/file3.rc
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=3/9)
.config/dir3/
.config/dir3/file4.rc
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=1/9)

sent 388 bytes  received 466 bytes  1,708.00 bytes/sec
total size is 0  speedup is 0.00

Even though this command successfully completed the task of ONLY copying over *.rc files along with the directories containing them, it did not delete the blubber and surplus files.

Seeing the "cannot delete non-empty directory: .config/surplus/waste/fat" like error messages, I naturally think that perhaps the "-m" (prune empty directories) option is preventing the rsync from getting rid of the unwanted directories, thus the next attempt:

_________________________________________
ATTEMPT 02:

Code:

rsync -avPp --include='*rc*' --include='*/' --exclude='*' --delete ./sourceHDD/.config ./destinationHDD
sending incremental file list
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus
cannot delete non-empty directory: .config/blubber
.config/
.config/file1.rc
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=7/9)
.config/file2.rc
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=6/9)
.config/dir1/
.config/dir1/dir2/
.config/dir1/dir2/file3.rc
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=1/9)
.config/dir3/
.config/dir3/file4.rc
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=0/9)
.config/dir5/

sent 414 bytes  received 478 bytes  1,784.00 bytes/sec
total size is 0  speedup is 0.00

Overall, this is not as successful as Attempt 01 since dir5 was copied over. And the "cannot delete non-empty directory: " error messages are still there. Unfortunately I think this could mean that the "--include='*/'" condition protects non-empty directories.

_________________________________________
ATTEMPT 03:
I then read the above link and scrolled down to "guettli"'s answer: '

Code:

find ./sourceHDD/.config -iname '*.rc' -print0 | rsync -0 -v --files-from=- . ./destinationHDD
AND 
cd ./sourceHDD/.config
find . -iname '*.rc' -print0 | rsync -0 -v --files-from=- . /home/${USER}/workshop/destinationHDD/.config

but these attempts don't copy to the target directory.

other failures

Code:

find ./source -iname '*.rc' -print0 | rsync -avPpm ./source/ ./destination/
{fail: this copies everything (*.txt and *.rc) into destination directory.}

find ./source/ -iname '*.rc' -print0 |  rsync -0 -v --files-from=- ./destination
{fail: rsync error}

Does anyone know how to run the command and then get rid of the non-empty directories not containing files with names containing "rc"?

nodir · 01-13-2017, 10:21 PM

Shooting ideas, and only that ! Take care
What about the approach to first delete all directories and files which don't contain '*.rc', then rsync the rest?

Something like:

Code:

find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d

to list all directories not containing rc in their name

Code:

find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d -exec rm -r {} +

to delete them
and

Code:

find sourceHDD/ ! -name '*.rc' -type f

to list files and

Code:

find sourceHDD/ ! -name '*.rc' -type f -exec rm {} +

to delete them.
Check if all is ok:

Code:

find sourceHDD ! -name '*rc'

Then rsync what is left.

I can see that it doesn't really hit the nail, lacks some cases. It is only an idea you might use for getting it sorted (or don't, if it doesn't suit your needs).
Someone with the according "find" knowledge should be able to give you better solutions, i guess.

andrew.comly · 01-14-2017, 10:42 PM

Quote:

Originally Posted by nodir

find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d
2) to list all directories not containing rc in their name

This would be good if I were looking for directories with names containing '*.rc', however I am looking for directories containing files ending in "rc".

It seems that I originally wrote the last sentence to vaguely: "Does anyone know how to run the command and then get rid of the non-empty directories not containing "*.rc"?"

I corrected it just now to "Does anyone know how to run the command and then get rid of the non-empty directories not [bold]containing files with names [/bold] containing "rc"?"

astrogeek · 01-15-2017, 02:45 AM

If I understand correctly...

Using your example directory structure, I have the following under some common parent directory which you call /home/${USER}/workshop/...

Code:

sourceHDD/
└── .config
    ├── blah.txt
    ├── dir1
    │** └── dir2
    │**     ├── file3.rc
    │**     └── file3.txt
    ├── dir3
    │** ├── file4.rc
    │** └── file4.txt
    ├── dir5
    │** └── new.txt
    ├── file1.rc
    ├── file1.txt
    ├── file2.rc
    └── file2.txt

destinationHDD/
└── .config
    ├── blubber
    │** └── trash1.txt
    └── surplus
        └── waste
            └── fat
                └── trash2.txt

You then want to copy all files named *.rc and their paths beginning with .config/ from the sourceHDD/ directory to the destinationHDD/, and remove all other files and directories in the process.

I would suggest simply removing everything below .config/ in the destination first, then use a simple find with exec to copy the desired structure from the source.

Code:

#!/bin/sh
rm -rf ./destinationHDD/.config/*
(cd sourceHDD/.config/
find . -iname '*rc' -exec cp --parent \{\} ../../destinationHDD/.config/ \;
)

Then with the above script in /home/${USER}/workshop/

Code:

./sync.sh

tree -Ra destinationHDD/
destinationHDD/
└── .config
    ├── dir1
    │** └── dir2
    │**     └── file3.rc
    ├── dir3
    │** └── file4.rc
    ├── file1.rc
    └── file2.rc

4 directories, 4 files

Is that the resultant tree you desire?

GazL · 01-15-2017, 05:07 AM

If I've understood your requirements correctly, then:

Code:

rsync --dry-run -v -a -p -m --include '*/' --include '*.rc' --exclude '*' --delete --delete-excluded sourceHDD/ destinationHDD

Remove the '--dry-run' only when you're certain it does what you want it to, rsync is a tricky beast!

Note: Unlike some unix commands, the trailing '/' on the source has a specific meaning to rsync. Don't omit it or you will end up with a './destinationHDD/sourceHDD/...' hierarchy instead of what you want.

andrew.comly · 01-16-2017, 11:40 PM

Quote:

Originally Posted by GazL

If I've understood your requirements correctly, then:

Code:

rsync --dry-run -v -a -p -m --include '*/' --include '*.rc' --exclude '*' --delete --delete-excluded sourceHDD/ destinationHDD

...

Outstanding answer, I found the key to your answer "--delete-excluded" in rsync manual pages file line 489. The fact that you have to first write "--delete" and then again "--delete-excluded" again right afterward never would have occurred to me.

This is EXACTLY what I was looking for, THANK-YOU SO MUCH!!!

P.S.- I'll try to use "--dry-run" more, that does save the acts of having to re mkdir -p and re-touch when using experimental directories for testing.

GazL · 01-17-2017, 05:46 AM

You're welcome.

BTW, in my example above, the '-p' is redundant as it's included by the '-a'. I missed that the first time around.