bash: recursive find, create parent folder, moving file

moocan · 12-05-2010, 06:18 PM

Hi,

I'm starting bash shell script and I'm looping without any solution.

I'm trying to find some files under a folder hierarchy and in case of errors moving these files to a destination folder under the same hierarchy recreating this hierarchy if not exists.

Finding all ._* files under /src and moving them to /dest recreating folder1 or the others which contains ._* files but without moving files which does not correspond to the pattern.

Code:

/src
 ._file1             -> move it to /dest
 ._file2             -> move it to /dest
 fileX               -> nothing to do
/src/folder1         -> recreate folder under /dest
 ._file3             -> move it to /dest/folder1
 ._file4             -> move it to /dest/folder1
 fileY               -> nothing to do
/src/folder1/folder4 -> recreate folder under /dest
 ._file5             -> move it to /dest/folder1/folder4
 fileB               -> nothing to do
/src/folder2         -> nothing to do
 fileD               -> nothing to do
...

as ..

Code:

/dest
 ._file1
 ._file2
/dest/folder1
 ._file3
 ._file4
/dest/folder1/folder4
 ._file5
...

I tried find command and I'am getting all needed files

Code:

find ./src/* -type f -name "._*"

But I don't know how to use the output to get the parent folder of files which are found to
1- create folder with mkdir -p /dest/folder1 or /dest/folder1/folder4
2- move found files from /src/... to /dest/... with rm command

I'm working on a find command as this trying to do all in the same line but ... little lost

Code:

find ./src/* -type file -name "._*" -exec dirname "{}" \; -exec basename "{}" \;

If someone could help me thanks in advance.

Kind Regards

grail · 12-05-2010, 06:41 PM

Does it have to be all on one line? Probably easier to break into a loop (using your find still to get the information) and then you can
either use parameter substitution or printf options for find to get the details you need to create folders and copy files.

crts · 12-05-2010, 10:13 PM

Hi,

try this:

Code:

find ./src/ -type f -name '._*' -exec sh -c 'mkdir -p "dest/$(dirname "${1#./src/}")";mv "${1}" "dest/${1#./src/}"' '{}' '{}' \;

moocan · 12-16-2010, 04:32 PM

Quote:

Originally Posted by grail

Does it have to be all on one line? Probably easier to break into a loop (using your find still to get the information) and then you can
either use parameter substitution or printf options for find to get the details you need to create folders and copy files.

I apologize for the long delay
As I have so many many folders within folders within folders on a NAS I'm trying another approach and I'm opened to all solutions.

I'm trying to do something like this too:

Code:

for folder in `ls -a -p ./src/`
if [ -d $folder ]
   folderlevel1 = `ls -a -p ./src/$folder`
   for folder in folderlevel1
     if [ -d $folderlevel1 ]
     folderlevel2 = `ls -a -p ./src/$folder/$folderlevel1`
       for each folder folderlevel2
         etc for each folderlevelX
           if searched files or dir are found
           srcfolder = src/folderlevel1/folderlevel2/folderlevelX/
           dstfolder = dst/folderlevel1/folderlevel2/folderlevelX/
           if $dstfolder does not exists
           mkdir $dstfolder
           mv found items from $srcfolder to $dstfolder
           cd ..
...... and so on within all folder or files structure.

I don't know how many folder levels could exists under a folder.
I'm trying to do something recursive.
I don't know how to control folders level without writing for loops many many many times and how to control the return back into the previous loop to handle next folders and so on within a complex folder hierarchy without for example restarting from previous folder always the same one.

Kind Regards

unSpawn · 12-16-2010, 04:45 PM

Moving something from src/ to dst/ in a --recursive way (implies folder creation anyway) and files to --exclude is typically something for 'rsync'. The only thing you need to be able to do is come up with --exclude regexes and --dry-run it before you execute it.

moocan · 12-17-2010, 08:48 AM

Quote:

Originally Posted by unSpawn

Moving something from src/ to dst/ in a --recursive way (implies folder creation anyway) and files to --exclude is typically something for 'rsync'. The only thing you need to be able to do is come up with --exclude regexes and --dry-run it before you execute it.

Many thanks to this solution.
I believe I can fly .....

I have tried many rsync command line and it works at 99%.
I have some folders which could be empty but don't want to sync all empty folders.

Here is my command line for ._* files and ._*/ folders but with --prune-empty-dirs if I use a protect filter (-f 'P ._*/) empty ._*/ folders are included but not deleted from sources and if I use Show & Risk filter (-f 'S ._*/,R ._*/') empty ._*/ folders are not included and so not deleted and if I remove --prune-empty-dirs all empty dirs are created.

Code:

rsync -aF -v --ignore-existing -f 'P ._*/' -f 'S ._*/,R ._*/' --include '._*' --include '._*/' --exclude '*.*' --prune-empty-dirs --verbose --dry-run --remove-sent-files ./src/ ./dst

Do you know any solution to include only some empty folders which correspond to a specific pattern which could then be deleted from source without syncing all empty dirs ?

EDIT1: rsync does not remove empty folders in source repository.

Kind Regards

unSpawn · 12-17-2010, 09:08 AM

Nice command line you whipped up there. I don't know how to selectively include empty dirs but maybe somebody else will. OTOH just recreating the empty file tree could be done easily afterwards using any other method, right?..

moocan · 12-18-2010, 01:48 PM

Hi,

It seems that I have found an alternative solution:

Code:

#!/bin/sh
StartingIn="./src"
MovingTo="./dst"

SAVEIFS=$IFS
IFS=$'\n'

items=`find "$StartingIn" -type file -name "._*"`

for item in $items
do
  echo "parsing: $item"
  
  fullname=${item#$StartingIn}
  filename=`basename "$fullname"`
  pathname=${fullname%$filename}
  
  if [ ! -d "$MovingTo$pathname" ]; then
  	mkdir -p "$MovingTo$pathname"
  fi
  
  mv -f -v "$item" "$MovingTo$pathname"

done

IFS=$SAVEIFS

I have some amelioration to add (write to log or read searched paterns from a config file) but I think that the essential is here.

I want to thanks each of you for your help.
Many thanks

Kind Regards

grail · 12-19-2010, 12:16 AM

I am also curious about your sometimes use of parameter substitution and also what you are looking to get from it??

Code:

  fullname=${item#$StartingIn}
  filename=`basename "$fullname"`
  pathname=${fullname%$filename}

So it appears you do all of this work to eventually come up with 'pathname' variable. Why not just stay with substitution:

Code:

  pathname=${item#$StartingIn}
  pathname=${pathname%/*}