LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 01-12-2017, 11:50 PM   #1
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 308
Blog Entries: 2

Rep: Reputation: 16
Question rsync: cannot delete non-empty directory


I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync.

PURPOSE:
1) Copy over all files/directories containing files ending in ".rc" FROM sourceHDD/.config TO destination/HDD/.config; (*.rc is of type file, not of type directory)
2) Delete any directory not recursively containing a file "*.rc" (not only empty directories).
3) Delete any file not containing a file "*.rc".

SITUATION:
I have a directory /home/${USER}/workshop/sourceHDD/.config, its file directory structure:
Code:
cd /home/${USER}/workshop
mkdir -p ./sourceHDD/.config/{dir1/dir2,dir3,dir5}
touch ./sourceHDD/.config/{file{1,2}.{txt,rc},dir3/file4.{rc,txt},dir1/dir2/file3.{rc,txt},dir5/new.txt,blah.txt}
I have a second directory /home/${USER}/workshop/destinationHDD/.config of the following file directory structure:
Code:
mkdir -p ./destinationHDD/.config/{blubber,surplus/waste/fat}
touch ./destinationHDD/.config/{blubber/trash1.txt,surplus/waste/fat/trash2.txt}
_________________________________________
ATTEMPT 01:
cd /home/${USER}/workshop
COMMAND:
Code:
rsync -avPpm --include='*rc*' --include='*/' --exclude='*' --delete ./source/* ./destination
RESULT:
Code:
9 files to consider
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus
cannot delete non-empty directory: .config/blubber
.config/
.config/file1.rc
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=7/9)
.config/file2.rc
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=6/9)
.config/dir1/
.config/dir1/dir2/
.config/dir1/dir2/file3.rc
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=3/9)
.config/dir3/
.config/dir3/file4.rc
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=1/9)

sent 388 bytes  received 466 bytes  1,708.00 bytes/sec
total size is 0  speedup is 0.00
Even though this command successfully completed the task of ONLY copying over *.rc files along with the directories containing them, it did not delete the blubber and surplus files.

Seeing the "cannot delete non-empty directory: .config/surplus/waste/fat" like error messages, I naturally think that perhaps the "-m" (prune empty directories) option is preventing the rsync from getting rid of the unwanted directories, thus the next attempt:

_________________________________________
ATTEMPT 02:
Code:
rsync -avPp --include='*rc*' --include='*/' --exclude='*' --delete ./sourceHDD/.config ./destinationHDD
sending incremental file list
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste/fat
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus/waste
cannot delete non-empty directory: .config/surplus
cannot delete non-empty directory: .config/blubber
.config/
.config/file1.rc
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=7/9)
.config/file2.rc
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=6/9)
.config/dir1/
.config/dir1/dir2/
.config/dir1/dir2/file3.rc
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=1/9)
.config/dir3/
.config/dir3/file4.rc
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=0/9)
.config/dir5/

sent 414 bytes  received 478 bytes  1,784.00 bytes/sec
total size is 0  speedup is 0.00
Overall, this is not as successful as Attempt 01 since dir5 was copied over. And the "cannot delete non-empty directory: " error messages are still there. Unfortunately I think this could mean that the "--include='*/'" condition protects non-empty directories.

_________________________________________
ATTEMPT 03:
I then read the above link and scrolled down to "guettli"'s answer: '
Code:
find ./sourceHDD/.config -iname '*.rc' -print0 | rsync -0 -v --files-from=- . ./destinationHDD
AND 
cd ./sourceHDD/.config
find . -iname '*.rc' -print0 | rsync -0 -v --files-from=- . /home/${USER}/workshop/destinationHDD/.config
but these attempts don't copy to the target directory.

other failures
Code:
find ./source -iname '*.rc' -print0 | rsync -avPpm ./source/ ./destination/
{fail: this copies everything (*.txt and *.rc) into destination directory.}

find ./source/ -iname '*.rc' -print0 |  rsync -0 -v --files-from=- ./destination
{fail: rsync error}
Does anyone know how to run the command and then get rid of the non-empty directories not containing files with names containing "rc"?

Last edited by andrew.comly; 01-14-2017 at 10:44 PM. Reason: Clarity
 
Old 01-13-2017, 10:21 PM   #2
nodir
Member
 
Registered: May 2016
Posts: 222

Rep: Reputation: Disabled
Shooting ideas, and only that ! Take care
What about the approach to first delete all directories and files which don't contain '*.rc', then rsync the rest?

Something like:
Code:
find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d
to list all directories not containing rc in their name
Code:
find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d -exec rm -r {} +
to delete them
and
Code:
find sourceHDD/ ! -name '*.rc' -type f
to list files and
Code:
find sourceHDD/ ! -name '*.rc' -type f -exec rm {} +
to delete them.
Check if all is ok:
Code:
find sourceHDD ! -name '*rc'
Then rsync what is left.


I can see that it doesn't really hit the nail, lacks some cases. It is only an idea you might use for getting it sorted (or don't, if it doesn't suit your needs).
Someone with the according "find" knowledge should be able to give you better solutions, i guess.
 
1 members found this post helpful.
Old 01-14-2017, 10:42 PM   #3
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 308

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Wink *.rc refers to a filename, not DIR name

Quote:
Originally Posted by nodir View Post
find sourceHDD/ -mindepth 2 ! -name '*.rc' -type d
2) to list all directories not containing rc in their name
This would be good if I were looking for directories with names containing '*.rc', however I am looking for directories containing files ending in "rc".

It seems that I originally wrote the last sentence to vaguely: "Does anyone know how to run the command and then get rid of the non-empty directories not containing "*.rc"?"

I corrected it just now to "Does anyone know how to run the command and then get rid of the non-empty directories not [bold]containing files with names [/bold] containing "rc"?"
 
Old 01-15-2017, 02:45 AM   #4
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,434
Blog Entries: 11

Rep: Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398Reputation: 3398
If I understand correctly...

Using your example directory structure, I have the following under some common parent directory which you call /home/${USER}/workshop/...

Code:
sourceHDD/
└── .config
    ├── blah.txt
    ├── dir1
    │** └── dir2
    │**     ├── file3.rc
    │**     └── file3.txt
    ├── dir3
    │** ├── file4.rc
    │** └── file4.txt
    ├── dir5
    │** └── new.txt
    ├── file1.rc
    ├── file1.txt
    ├── file2.rc
    └── file2.txt

destinationHDD/
└── .config
    ├── blubber
    │** └── trash1.txt
    └── surplus
        └── waste
            └── fat
                └── trash2.txt
You then want to copy all files named *.rc and their paths beginning with .config/ from the sourceHDD/ directory to the destinationHDD/, and remove all other files and directories in the process.

I would suggest simply removing everything below .config/ in the destination first, then use a simple find with exec to copy the desired structure from the source.

Code:
#!/bin/sh
rm -rf ./destinationHDD/.config/*
(cd sourceHDD/.config/
find . -iname '*rc' -exec cp --parent \{\} ../../destinationHDD/.config/ \;
)
Then with the above script in /home/${USER}/workshop/

Code:
./sync.sh

tree -Ra destinationHDD/
destinationHDD/
└── .config
    ├── dir1
    │** └── dir2
    │**     └── file3.rc
    ├── dir3
    │** └── file4.rc
    ├── file1.rc
    └── file2.rc

4 directories, 4 files
Is that the resultant tree you desire?

Last edited by astrogeek; 01-15-2017 at 02:46 AM.
 
1 members found this post helpful.
Old 01-15-2017, 05:07 AM   #5
GazL
LQ Guru
 
Registered: May 2008
Posts: 5,540
Blog Entries: 14

Rep: Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382
If I've understood your requirements correctly, then:
Code:
rsync --dry-run -v -a -p -m --include '*/' --include '*.rc' --exclude '*' --delete --delete-excluded sourceHDD/ destinationHDD
Remove the '--dry-run' only when you're certain it does what you want it to, rsync is a tricky beast!


Note: Unlike some unix commands, the trailing '/' on the source has a specific meaning to rsync. Don't omit it or you will end up with a './destinationHDD/sourceHDD/...' hierarchy instead of what you want.
 
1 members found this post helpful.
Old 01-16-2017, 11:40 PM   #6
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 308

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Talking [solved]

Quote:
Originally Posted by GazL View Post
If I've understood your requirements correctly, then:
Code:
rsync --dry-run -v -a -p -m --include '*/' --include '*.rc' --exclude '*' --delete --delete-excluded sourceHDD/ destinationHDD
...
Outstanding answer, I found the key to your answer "--delete-excluded" in rsync manual pages file line 489. The fact that you have to first write "--delete" and then again "--delete-excluded" again right afterward never would have occurred to me.

This is EXACTLY what I was looking for, THANK-YOU SO MUCH!!!

P.S.- I'll try to use "--dry-run" more, that does save the acts of having to re mkdir -p and re-touch when using experimental directories for testing.

Last edited by andrew.comly; 01-16-2017 at 11:44 PM. Reason: grammar
 
1 members found this post helpful.
Old 01-17-2017, 05:46 AM   #7
GazL
LQ Guru
 
Registered: May 2008
Posts: 5,540
Blog Entries: 14

Rep: Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382Reputation: 3382
You're welcome.


BTW, in my example above, the '-p' is redundant as it's included by the '-a'. I missed that the first time around.
 
1 members found this post helpful.
  


Reply

Tags
dir, exclude, include, rsync


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
BTRFS cannot delete Directory not empty unkie888 Linux - Software 1 10-22-2012 05:08 PM
[SOLVED] Cannot delete directory: non empty because a crazy symlink rfsleal Programming 3 03-11-2012 02:17 PM
Can't remove non-empty directory with rsync alkos333 Slackware 5 01-20-2009 06:31 PM
Delete Empty files whospeaks2me Linux - Newbie 2 07-18-2003 10:39 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:17 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration