Collate all files from multiple folders

LQ496873 · 03-05-2022, 05:51 AM

I've got about 30 folders each containing a few hundred files. I want copy all of the files in all of the folders into a new folder.

I've been trying to use rsync but I'm not sure it can do it. Internet searches keep resulting in programming scripts and the obvious...

Code:

cp my_file_name.txt my_file2_name.txt

Can it be done with a somewhat simple command?

Thank you for any help

Turbocapitalist · 03-05-2022, 05:59 AM

More information is needed but you can do that with the perl version of rename from the 'rename' package. Or else find might be needed, too. How do you plan to calculate the new name?

LQ496873 · 03-05-2022, 06:41 AM

I did consider using the mv command but like with cp it seems you have to specify every file. I'm unfamiliar with perl and rename... and running scripts too! My only programming has been with VSCode and Python.

Essentially, I've used photorec on an old 3GB IDE drive I found in the garage and I want to put all the files into one folder. I'm not bothered about the names of them because photorec has given them all obscure names already.

computersavvy · 03-05-2022, 08:58 AM

I have done similar when I ripped some audio book CDs onto my PC. Each CD was written into its own folder with the same series of file names and I wound up writing a script that looped through and read the file names then renamed them into one folder with a continuous series of names so the names stayed in the order they would be listened to.

It would be very simple to rename them something like folder-filename sequence, since the folders are likely already unique names.

syg00 · 03-05-2022, 04:42 PM

Quote:

Originally Posted by LQ496873

I did consider using the mv command but like with cp it seems you have to specify every file.

Nope - but you will have to specify each of the source directories. If files are mv'd in the same filesystem, there is no copying of the data involved, just an update of the inode. "ls -d */" will give you a list of dirs - may or may not need filtering; try it and see. This should work for the move - "mv dir1/* target"

Sefyir · 03-05-2022, 11:49 PM

Are the filenames unique and able to just be copied into the new directory?

If your photorec is at /data and new directory is /flat

Code:

find /data -type f -exec cp {} /flat \;

If it's on the same drive and you want to ensure you don't overwrite files

Code:

find /data -type f -exec cp --no-clobber --link {} /flat \;

This scans /data for everything with type f (normal file), then executes a cp of that file to /flat

LQ496873 · 03-06-2022, 05:02 AM

Quote:

Originally Posted by Sefyir

Are the filenames unique and able to just be copied into the new directory?

If your photorec is at /data and new directory is /flat

Code:

find /data -type f -exec cp {} /flat \;

If it's on the same drive and you want to ensure you don't overwrite files

Code:

find /data -type f -exec cp --no-clobber --link {} /flat \;

This scans /data for everything with type f (normal file), then executes a cp of that file to /flat

Thank you so much for your help guys... this worked perfectly. I've had many bouts of searching for the answer to no avail, despite the question being asked many times on other websites. I hope this thread can help others in the future.

LQ496873 · 03-11-2022, 04:53 AM

I've decided to go through the same process with all the other drives I have kicking about. I've accumulated about 25 over the years!
Some are pretty big so I've added to the script so that instead of dumping all the files into one folder they are separated by file type... ie /jpg, /png, /rar etc

The downside is that I have to return to the computer, change "filetype=***" and restart the script. I don't mind restarting it for each drive but it's quite repetitive doing it for each file type on each drive!

Is there a way to change what I've done so far so that "filetype=***" is changed for each filetype in /data until everything has been sorted and /data is deleted?
A bit like... "for i in /data, filetype=i"

Code:

filetype=png 
&& original=data 
&& new=flat 
&& mkdir -p /$new/$filetype/ 
&& find /$original -name "*.$filetype" -type f -exec cp -v {} /$new/$filetype \; 
&& sudo find /$original -name "*.$filetype" -type f -exec rm -v {} \; 
&& find /$original -empty -type d -delete

Thank you!

computersavvy · 03-11-2022, 09:46 AM

Of course it can handle more than one path and file type.

Put a loop within a loop so that the outer loop selects the file type and path for output and the inner loop does the processing for the files that match.

LQ496873 · 03-12-2022, 02:04 AM

Thank you. I think I need to learn scripting properly! I can do a bit of Python and thought about coding it all into a GUI but maybe this is my perfect reason to learn Linux better.

I can use

Code:

find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

to get a list of file extensions that photorec salvaged but getting my script to loop it is proving difficult.

Should I be saving my script as a .sh file and running that? At the moment I'm just running a really long command in the terminal... (the last one being)

Code:

filetype=doc && original=/run/media/richard/8b491c4c-3477-4e22-9ecf-b8492d060e49/80gbo && new=/run/media/richard/8b491c4c-3477-4e22-9ecf-b8492d060e49/80gbBlack && mkdir -p $new/$filetype/ && find $original -name "*.$filetype" -type f -exec cp -v {} $new/$filetype \; && sudo find $original -name "*.$filetype" -type f -exec rm -v {} \; && find $original -empty -type d -delete

I feel I've tried a hunderd versions of things like

Code:

for i in {find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u}
do 
filetype=$i && original=/run/media/richard/8b491c4c-3477-4e22-9ecf-b8492d060e49/80gbo && new=/run/media/richard/8b491c4c-3477-4e22-9ecf-b8492d060e49/80gbBlack && mkdir -p $new/$filetype/ && find $original -name "*.$filetype" -type f -exec cp -v {} $new/$filetype \; && sudo find $original -name "*.$filetype" -type f -exec rm -v {} \; && find $original -empty -type d -delete

That type of thing. Obvioulsy stright away list=... doesn't work.

MadeInGermany · 03-12-2022, 06:44 AM

The following hooks an embedded bash script into a find.

Code:

cd "$original"
find . -type f -exec /bin/bash -c '
  for f
  do
    if [[ $f == *.?* ]]
    then extd=${f##*.}
    else extd=noext
    fi
    mkdir -p "$new/$extd" &&
    cp "$f" "$new/$extd/$f" &&
    sudo rm "$f"
  done
' move.sh {} +