LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 05-26-2009, 01:23 PM   #1
daberkow
LQ Newbie
 
Registered: May 2009
Posts: 11

Rep: Reputation: 0
bash scripting: sorting and using using multiple files in a script


Hi. I have a directory full of files that are named as such:

yyyy_mm_dd_ID_more_stuff.ext

For each distinct ID, there are two files, and these two files have different date prefixes- ie for AA I have:

2009_01_01_AA_stuff.ext
2009_03_22_AA_stuff.ext

What I want to do is have a script that will loop through the IDs and in each iteration call the 'earlier' file and the 'later' file in the correct order. Something like:
Code:
command -input1 ${firstfile} -input2 ${secondfile} -output ${ID}_output
To sort out the IDs, I have been using
Code:
ls | awk -F_ '{print $4}'
which gives me a list of IDs I can for loop through, but from here I am stuck. Am I even going about this the right way?

How would (or should) I assign each individual file containing the given ID to a variable I can call? The files are named in order by date so a simple ls returns the correct order once I have them separated by ID.

Thanks.
 
Old 05-26-2009, 04:36 PM   #2
crabboy
Moderator
 
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,823

Rep: Reputation: 120Reputation: 120
I think you're on the right track. To take a quick stab at it, I'd probably do something like this:
Code:
#!/bin/sh

for i in `ls | awk -F_ ' { print $4 }' | sort -u`; do
   FILES=`ls | grep $i`
   FILE1=`echo $FILES | cut -d' ' -f1`
   FILE2=`echo $FILES | cut -d' ' -f2`
   echo "command -input 1 ${FILE1} -input2 ${FILE2}"
done
 
Old 05-26-2009, 07:28 PM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,311

Rep: Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040
I'd add another sort to get them in date order

FILES=`ls | grep $i |sort`
 
Old 05-26-2009, 08:46 PM   #4
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Code:
ls | awk 'BEGIN{ FS="_" }
{
     t=$1$2$3 
     if ( $4 in id ){
        if ( t >= id[$4] ){
            file[$4] =file[$4]" "$0            
        } 
     }else{
        file[$4]=file[$4]" "$0
        id[$4]=t
     }
}  
END{
 for( i in file){
    m=split(file[i],f," ")
    cmd = "command -input1 "f[1]" -input2 "f[2]" -output "i"_output"
    print cmd
    #system(cmd) #uncomment to use
 }
}'
output
Code:
# ls -1
2009_01_01_AA_stuff.ext
2009_01_02_BB_stuff.ext
2009_01_05_BB_stuff.ext
2009_01_05_DD_stuff.ext
2009_01_30_DD_stuff.ext
2009_03_05_CC_stuff.ext
2009_03_22_AA_stuff.ext
2009_03_22_CC_stuff.ext

#./test.sh
command -input1 2009_01_01_AA_stuff.ext -input2 2009_03_22_AA_stuff.ext -output AA_output
command -input1 2009_01_02_BB_stuff.ext -input2 2009_01_05_BB_stuff.ext -output BB_output
command -input1 2009_03_05_CC_stuff.ext -input2 2009_03_22_CC_stuff.ext -output CC_output
command -input1 2009_01_05_DD_stuff.ext -input2 2009_01_30_DD_stuff.ext -output DD_output
 
Old 05-26-2009, 08:50 PM   #5
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by crabboy View Post
I think you're on the right track. To take a quick stab at it, I'd probably do something like this:
Code:
#!/bin/sh

for i in `ls | awk -F_ ' { print $4 }' | sort -u`; do
   FILES=`ls | grep $i`
   FILE1=`echo $FILES | cut -d' ' -f1`
   FILE2=`echo $FILES | cut -d' ' -f2`
   echo "command -input 1 ${FILE1} -input2 ${FILE2}"
done
Code:
for i in `ls | awk -F_ ' { print $4 }' | sort -u`; do
   FILES=`ls | grep $i`
too much overhead.
 
Old 05-26-2009, 08:52 PM   #6
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by chrism01 View Post
I'd add another sort to get them in date order

FILES=`ls | grep $i |sort`
no need grep
Code:
ls *${i}*
 
Old 05-27-2009, 12:11 PM   #7
daberkow
LQ Newbie
 
Registered: May 2009
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks! Everything works perfectly. The awk solution is a bit over my head... what do you mean by 'too much overhead' for the other case?

Quote:
Originally Posted by ghostdog74 View Post
Code:
for i in `ls | awk -F_ ' { print $4 }' | sort -u`; do
   FILES=`ls | grep $i`
too much overhead.
 
Old 05-27-2009, 12:27 PM   #8
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by daberkow View Post
what do you mean by 'too much overhead' for the other case?
if you have 1000 files currently, you are iterating over 1000 files, and then doing
Code:
FILES=`ls | grep $i`
1000 times. the above will do ls on 1000 files again.

Last edited by ghostdog74; 05-27-2009 at 12:33 PM.
 
Old 05-28-2009, 10:24 AM   #9
crabboy
Moderator
 
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,823

Rep: Reputation: 120Reputation: 120
yea, I'd agree that it is not optimal, but I choose it because it was easier than using an array.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Bash script for sorting and renaming multiple mp3 files by id3 tags simonloach Linux - General 8 02-16-2013 10:07 AM
Help with Bash Script - Rename Multiple Files embsupafly Programming 16 04-02-2010 04:50 AM
bash script: open multiple rar files chief_officer Programming 1 11-16-2007 03:04 AM
Sorting files in BASH deleted/ Linux - Newbie 16 01-26-2006 07:03 AM
Bash backup script - If multiple files starting with a exist problem demoncheese Programming 2 07-29-2004 11:47 PM


All times are GMT -5. The time now is 07:43 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration