LinuxQuestions.org - [SOLVED] Scripting to move thousands of files into folders

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Scripting to move thousands of files into folders (https://www.linuxquestions.org/questions/linux-newbie-8/scripting-to-move-thousands-of-files-into-folders-882594/)

Scripting to move thousands of files into folders

Our client has a website (Joomla 1.5 based) to promote participation in and report results of a 10k race for runners in Scandinavia...

The race is scheduled for late June and will be covered by at least 5 still photographers at various points along the race course.

There are to be about 5,000 runners, each entered under a shirt number like bib_1000 through bib_5999, and this means there will be a significant number of digital jpg files submitted by the photographers...

The client has in place some software which will "automagically" resize, watermark, and rename the individual files in the form:

bib_1234_1.jpg
bib_1234_2.jpg
bib_1234_3.jpg

and so on with a possibility of as many as 10 or 12 images in the aggregate with the same alpha-numeric "shirt number" at the beginning of the filename.

Our presentation software does a great job of handling thumbnail image generation and displaying a slideshow in a lightbox of all files within a given folder on the webserver.

and now finally the question...

Given a folder containing 25,000 or so *.jpg files, how can we write a script that:

1. parses the filenames to unique "shirt numbers"
2. makes folders with each "shirt number" as the foldername
3. moves the files from the root of the original folder into the appropriate "shirt number" folder.

Note that the order of my list above is not important, and if you know of a better way to organize the task we are fine with that.

Thanks for your patience in reading through this.

Hi,

try if this will do:

Code:

for f in *; do

 if [[ -f "${f}" ]]; then

  g="${f#*_}"

  mkdir "${g%_*}"

  mv "bib_${g%_*}"* "${g%_*}"

 fi

done

Execute it from inside the directory that holds all pictures.
I tested it with a bunch of dummy files which I created with

Code:

touch bib_{1234,2341,3412,4123}_{1..10}.jpg

Those were, of course, no real jpg-files. But that doesn't affect the program's logic.

Hi,

how does actual your script look like? Or are you looking for someone who will do the complete work?

All users here are volunteers, so if you ask some question you should put here what you actual have done. If you are looking for someone doing your work you better hire a freelancer and took the bill to your customer.

Definitly you should read LQ Rules.

crts .. your slipping ... could have used the 'f' again in the move so no need to type 'bib_' ... hehe .. just joshin :)

Good work as always

@grail: Thanks for the hint :)
Initially I had a more general match in mind, like
*"${g%_*}"*

but that backfired with the directory names. So I just changed it without giving it any further thought.
It should indeed be 'f'.

For illustrative purposes, this is what I came up with

Code:

#!/bin/sh



for bib in $( ls *.jpg | cut -f 2 -d _ | uniq ); do

    mkdir -p ./pictures/$bib

    find . -maxdepth 1 -name "bib_"$bib"_*.jpg" -exec mv {} ./pictures/$bib/ \;

done

Quote:

Originally Posted by hf2046 (Post 4366577)

For illustrative purposes, this is what I came up with

Code:

#!/bin/sh



for bib in $( ls *.jpg | cut -f 2 -d _ | uniq ); do

    mkdir -p ./pictures/$bib

    find . -maxdepth 1 -name "bib_"$bib"_*.jpg" -exec mv {} ./pictures/$bib/ \;

done

Hi,

I like the idea of feeding a uniq list to the for-loop. However, read this why it is not a good idea to parse ls.
Then read more why you should avoid the command substitution in this case. The 'find' command is also redundant.
So with some corrections:

Code:

stat -c %n *.jpg | cut -f 2 -d _ | uniq | while read bib; do

 mkdir -p "$bib"

 mv "bib_$bib"* "$bib"/

done

Quote:

Originally Posted by crts (Post 4366609)

Code:

stat -c %n *.jpg

You might as well just use "ls -1" (the number one, not lowercase letter "L").

EDIT: Anyway, here's my solution :):

Code:

for picture in *.jpg

do

        dir="${picture%_*}" # if picture is "bib_1234_1.jpg", dir will be "bib_1234"

        if [ '!' -e "${dir}" ]

        then

                mkdir "${dir}"

        fi

        mv "${picture}" "${dir}"

done

Quote:

Originally Posted by MTK358 (Post 4366643)

You might as well just use "ls -1" (the number one, not lowercase letter "L").

That is true and I have done so in the past. However, I am trying to make it a personal general habit to not use 'ls' in a script - even if it would be possible as in the aforementioned case.

Quote:

Originally Posted by crts (Post 4366609)

Code:

stat -c %n *.jpg | cut -f 2 -d _ | uniq | while read bib; do

 mkdir -p "$bib"

 mv "bib_$bib"* "$bib"/

done

That find was redundant - doh! Did not know about ls and command substitution... good stuff. Thanks for the links!

Well just to give the user another alternative:

Code:

#!/usr/bin/awk



BEGIN{

    while(ARGV[++i]){

        split(ARGV[i],n,"_")



        if(! (n[2] in dir)){

            dir[n[2]]++

            cmd = sprintf("mkdir %s;mv *%s*.jpg %s",n[2],n[2],n[2])



            print | cmd

            cmd = ""

        }

    }

}

And call it from within the directory with:

Code:

./script.awk *jpg

Thanks to all who contributed...

I must have erred when posting, because I did not receive emails regarding your posts, only just the one this am about whether or not the issue had been resolved.

I don't know the answer to that yet, but I believe your collective work and your reviews should prove successful.

Thank you all again.

Quote:

Originally Posted by mesiol (Post 4366430)

...

Definitly you should read LQ Rules.

Thanks for your response, I did review the forum rules, thinking it would be obvious which of them you believe I have violated, but it was not. Could you be more specific in your accusation?

I can certainly understand your annoyance if you thought I was asking you to do work for which I was charging my client, but that is not the case... my scope STARTS when the folders are full of images.