LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-25-2011, 06:02 AM   #1
ranwilli
LQ Newbie
 
Registered: May 2011
Location: Toledo, OH
Posts: 3

Rep: Reputation: Disabled
Scripting to move thousands of files into folders


Our client has a website (Joomla 1.5 based) to promote participation in and report results of a 10k race for runners in Scandinavia...

The race is scheduled for late June and will be covered by at least 5 still photographers at various points along the race course.

There are to be about 5,000 runners, each entered under a shirt number like bib_1000 through bib_5999, and this means there will be a significant number of digital jpg files submitted by the photographers...

The client has in place some software which will "automagically" resize, watermark, and rename the individual files in the form:

bib_1234_1.jpg
bib_1234_2.jpg
bib_1234_3.jpg

and so on with a possibility of as many as 10 or 12 images in the aggregate with the same alpha-numeric "shirt number" at the beginning of the filename.

Our presentation software does a great job of handling thumbnail image generation and displaying a slideshow in a lightbox of all files within a given folder on the webserver.

and now finally the question...

Given a folder containing 25,000 or so *.jpg files, how can we write a script that:

1. parses the filenames to unique "shirt numbers"
2. makes folders with each "shirt number" as the foldername
3. moves the files from the root of the original folder into the appropriate "shirt number" folder.

Note that the order of my list above is not important, and if you know of a better way to organize the task we are fine with that.

Thanks for your patience in reading through this.
 
Old 05-25-2011, 06:54 AM   #2
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Hi,

try if this will do:
Code:
for f in *; do
 if [[ -f "${f}" ]]; then
  g="${f#*_}"
  mkdir "${g%_*}"
  mv "bib_${g%_*}"* "${g%_*}"
 fi
done
Execute it from inside the directory that holds all pictures.
I tested it with a bunch of dummy files which I created with
Code:
touch bib_{1234,2341,3412,4123}_{1..10}.jpg
Those were, of course, no real jpg-files. But that doesn't affect the program's logic.
 
Old 05-25-2011, 06:55 AM   #3
mesiol
Member
 
Registered: Nov 2008
Location: Lower Saxony, Germany
Distribution: CentOS, RHEL, Solaris 10, AIX, HP-UX
Posts: 731

Rep: Reputation: 137Reputation: 137
Hi,

how does actual your script look like? Or are you looking for someone who will do the complete work?

All users here are volunteers, so if you ask some question you should put here what you actual have done. If you are looking for someone doing your work you better hire a freelancer and took the bill to your customer.

Definitly you should read LQ Rules.
 
Old 05-25-2011, 08:25 AM   #4
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,421

Rep: Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815
crts .. your slipping ... could have used the 'f' again in the move so no need to type 'bib_' ... hehe .. just joshin

Good work as always

Last edited by grail; 05-25-2011 at 08:31 AM.
 
Old 05-25-2011, 08:49 AM   #5
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
@grail: Thanks for the hint
Initially I had a more general match in mind, like
*"${g%_*}"*

but that backfired with the directory names. So I just changed it without giving it any further thought.
It should indeed be 'f'.
 
Old 05-25-2011, 09:57 AM   #6
hf2046
Member
 
Registered: Mar 2011
Distribution: Slack64
Posts: 111

Rep: Reputation: 20
For illustrative purposes, this is what I came up with

Code:
#!/bin/sh

for bib in $( ls *.jpg | cut -f 2 -d _ | uniq ); do
    mkdir -p ./pictures/$bib
    find . -maxdepth 1 -name "bib_"$bib"_*.jpg" -exec mv {} ./pictures/$bib/ \;
done
 
Old 05-25-2011, 10:32 AM   #7
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Quote:
Originally Posted by hf2046 View Post
For illustrative purposes, this is what I came up with

Code:
#!/bin/sh

for bib in $( ls *.jpg | cut -f 2 -d _ | uniq ); do
    mkdir -p ./pictures/$bib
    find . -maxdepth 1 -name "bib_"$bib"_*.jpg" -exec mv {} ./pictures/$bib/ \;
done
Hi,

I like the idea of feeding a uniq list to the for-loop. However, read this why it is not a good idea to parse ls.
Then read more why you should avoid the command substitution in this case. The 'find' command is also redundant.
So with some corrections:
Code:
stat -c %n *.jpg | cut -f 2 -d _ | uniq | while read bib; do
 mkdir -p "$bib"
 mv "bib_$bib"* "$bib"/
done
 
1 members found this post helpful.
Old 05-25-2011, 11:07 AM   #8
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
Quote:
Originally Posted by crts View Post
Code:
stat -c %n *.jpg
You might as well just use "ls -1" (the number one, not lowercase letter "L").

EDIT: Anyway, here's my solution :

Code:
for picture in *.jpg
do
	dir="${picture%_*}" # if picture is "bib_1234_1.jpg", dir will be "bib_1234"
	if [ '!' -e "${dir}" ]
	then
		mkdir "${dir}"
	fi
	mv "${picture}" "${dir}"
done

Last edited by MTK358; 05-25-2011 at 11:19 AM.
 
Old 05-25-2011, 11:21 AM   #9
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Quote:
Originally Posted by MTK358 View Post
You might as well just use "ls -1" (the number one, not lowercase letter "L").
That is true and I have done so in the past. However, I am trying to make it a personal general habit to not use 'ls' in a script - even if it would be possible as in the aforementioned case.
 
Old 05-25-2011, 12:45 PM   #10
hf2046
Member
 
Registered: Mar 2011
Distribution: Slack64
Posts: 111

Rep: Reputation: 20
Quote:
Originally Posted by crts View Post
Hi,

I like the idea of feeding a uniq list to the for-loop. However, read this why it is not a good idea to parse ls.
Then read more why you should avoid the command substitution in this case. The 'find' command is also redundant.
So with some corrections:
Code:
stat -c %n *.jpg | cut -f 2 -d _ | uniq | while read bib; do
 mkdir -p "$bib"
 mv "bib_$bib"* "$bib"/
done
That find was redundant - doh! Did not know about ls and command substitution... good stuff. Thanks for the links!
 
Old 05-26-2011, 03:23 AM   #11
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,421

Rep: Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815Reputation: 2815
Well just to give the user another alternative:
Code:
#!/usr/bin/awk

BEGIN{
    while(ARGV[++i]){
        split(ARGV[i],n,"_")

        if(! (n[2] in dir)){
            dir[n[2]]++
            cmd = sprintf("mkdir %s;mv *%s*.jpg %s",n[2],n[2],n[2])

            print | cmd
            cmd = ""
        }
    }
}
And call it from within the directory with:
Code:
./script.awk *jpg
 
Old 06-02-2011, 06:18 AM   #12
ranwilli
LQ Newbie
 
Registered: May 2011
Location: Toledo, OH
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks to all who contributed...

I must have erred when posting, because I did not receive emails regarding your posts, only just the one this am about whether or not the issue had been resolved.

I don't know the answer to that yet, but I believe your collective work and your reviews should prove successful.


Thank you all again.

Last edited by ranwilli; 06-02-2011 at 06:31 AM. Reason: edited for stupid typo
 
Old 06-02-2011, 06:30 AM   #13
ranwilli
LQ Newbie
 
Registered: May 2011
Location: Toledo, OH
Posts: 3

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by mesiol View Post
...

Definitly you should read LQ Rules.
Thanks for your response, I did review the forum rules, thinking it would be obvious which of them you believe I have violated, but it was not. Could you be more specific in your accusation?

I can certainly understand your annoyance if you thought I was asking you to do work for which I was charging my client, but that is not the case... my scope STARTS when the folders are full of images.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Write a script to move specific files in various folders to one folder linuxisthedevil Linux - General 13 11-18-2010 07:29 AM
I can't move my files to folders using metacharacters (mv ./.blah ../?[0-9]/) Cyberman Programming 5 06-29-2008 09:34 PM
Can't move files/folders to trash... adred Linux - General 1 06-22-2008 06:40 AM
Alternative to such good Flashfxp that can move the files/folders into the ftp ? frenchn00b Linux - Software 2 06-08-2008 11:45 PM
move files in tar.gz back to into original folders GUIPenguin Linux - General 1 05-09-2005 01:11 PM


All times are GMT -5. The time now is 08:06 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration