LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-01-2011, 08:01 AM   #1
umix
LQ Newbie
 
Registered: Oct 2011
Posts: 13

Rep: Reputation: Disabled
sorting of files and creating a loop to process them date wise


There is a sourcefile like:

AA_impact2010-04-15.log
BB_impact2010-04-14.log
CC_impact2010-04-14.log
DD_impact2010-04-13.log
EE_impact2010-04-13.log

I have to process the files datewise like

list of files arrived on 13th:
echo"start processing"(i have the processing script)

end
list of files arrived on 14th:
echo"start processing" (.script.ksh)

end

list of files arrived on 15th:
echo "start processing"(.script.ksh)

end


I am a newbee to unix so any suggestion will be helpful
thanks in advance :-)
 
Old 12-01-2011, 08:52 AM   #2
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,397
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
If your sourcefile is called LQumix.dat
Code:
#! /bin/bash
#
#   First, extract unique dates for sorting, and iterate over each 
#
for filedate in $(sed 's/.*_impact//' LQumix.dat | uniq | sort ); do 
    echo Processing files for date: ${filedate}
    #    
    #  Now, find files with matching dates in the names, and process them
    #
    for datafile in "*_impact${filedate}"; do
        echo "processing ${datafile}"
    done
done
---- rod

Last edited by theNbomr; 12-01-2011 at 05:07 PM. Reason: Fix typo and add code improvement
 
Old 12-01-2011, 09:18 AM   #3
umix
LQ Newbie
 
Registered: Oct 2011
Posts: 13

Original Poster
Rep: Reputation: Disabled
thank you theNbomr

but it solved my problem partially...
i am trying to process the files from a particular folder but it is taking the files from all the subfolders wherever it gets the date pattern inspite of mentioning the file name.
 
Old 12-01-2011, 10:31 AM   #4
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
You're lucky because the date format yyyy-mm-dd (as it appears in your file names) is one of the standard formats recognized by the date command. A loop over dates would appear similar to this:
Code:
#!/bin/bash
day=2010-04-13
while [[ $day != 2010-05-01 ]]
do
  echo ${day}
  day=$(date -d "$day 1 day" +%Y-%m-%d)
done
Inside the loop you can embed the ${day} referenced value into the wanted file names.
 
Old 12-01-2011, 10:43 AM   #5
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,397
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Quote:
Originally Posted by umix View Post
thank you theNbomr

but it solved my problem partially...
i am trying to process the files from a particular folder but it is taking the files from all the subfolders wherever it gets the date pattern inspite of mentioning the file name.
Okay, then just remove the 'find -name' and the enclosing command substitution parentheses from the loop iterator set in the inner loop.
Code:
    for datafile in "*${filedate}"; do
--- rod.
 
Old 12-01-2011, 04:25 PM   #6
makyo
Member
 
Registered: Aug 2006
Location: Saint Paul, MN, USA
Distribution: {Free,Open}BSD, CentOS, Debian, Fedora, Solaris, SuSE
Posts: 729

Rep: Reputation: 75
Hi.

One solution is to use standard and available utilities. For example, this script displays my context, then an augmented data file, and then results:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate sorting of y-m-d embedded field with break, split.
# http://www1.cuni.cz/~obo/textutils/
# http://freecode.com/projects/msort

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C msort split_at_colchange

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

# Remove temporary files.
rm f1 f2
pl " Results:"
sed 's/\(^.*impact\)/\1\t/;s/\./\t/' $FILE |
tee f1 |
msort --quiet --line --position 2 --comparison-type d --date-format "y-m-d" |
tee f2 |
split_at_colchange 2 |
sed 's/\t//;s/\t/\./'

exit 0
orudcing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.32-5-686, i686
Distribution        : Debian GNU/Linux 6.0.3 (squeeze) 
GNU bash 4.1.5
msort 8.53
split_at_colchange - ( local: RepRev 1.2, ~/bin/split_at_colchange, 2011-12-01 )

-----
 Input data file data1:
AA_impact2010-03-15.log
AA_impact2010-05-15.log
AA_impact2010-04-15.log
BB_impact2010-04-14.log
CC_impact2010-04-14.log
DD_impact2010-04-13.log
EE_impact2010-04-13.log
DD_impact2009-04-13.log

-----
 Results:
DD_impact2009-04-13.log

AA_impact2010-03-15.log

DD_impact2010-04-13.log
EE_impact2010-04-13.log

BB_impact2010-04-14.log
CC_impact2010-04-14.log

AA_impact2010-04-15.log

AA_impact2010-05-15.log
The msort utility was in my Debian repository (as well as in freecode). The perl utility split_at_colchange is available at the site mentioned in the script comment.

The script uses sed to effectively split apart the data so that the date is the second field. This is piped forward (and copied to temporary file f1 for viewing). The sort orders lines based on the dates in the second field (copied to temporary f2), and then data lines are re-assembled. From here, commands like csplit could be used to create separate files of lines of date clusters based on the occurrence of an empty line if desired.

The msort will create a file msort.log containing statistics about the sorting operation.


Best wishes ... cheers, makyo
 
Old 12-02-2011, 01:38 AM   #7
umix
LQ Newbie
 
Registered: Oct 2011
Posts: 13

Original Poster
Rep: Reputation: Disabled
@ theNbomr

i tried following but still not working:
for datafile in "*${filedate}"; do

it is not taking the files from that particular folder...
 
Old 12-02-2011, 11:40 AM   #8
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,397
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Quote:
Originally Posted by umix View Post
@ theNbomr

i tried following but still not working:
for datafile in "*${filedate}"; do

it is not taking the files from that particular folder...
The following is a filespec expression for 'all the files with names ending with the value of $filedate, and that exist in the current working directory'. The value of $filedate will vary depending on what it reads from the sourcefile. There is an implied directory component of the 'current working directory'. It sounds like you need a specific directory.
Code:
*${filedate}
How does the filespec expression differ from what you want? If you can answer that question in words, I feel certain that you will arrive at the solution by yourself.

--- rod.
 
Old 12-03-2011, 04:12 AM   #9
umix
LQ Newbie
 
Registered: Oct 2011
Posts: 13

Original Poster
Rep: Reputation: Disabled
Thank you all for your valuable suggestions..

It helped me a lot..
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Date wise Folder name opi045 Linux - Software 1 01-21-2009 12:13 PM
Squid Access Logs date wise the_gripmaster Linux - Software 1 10-03-2007 09:38 PM
Sorting all files of home folder modified after a specific date for backup fc6_user Linux - General 2 03-30-2007 02:24 AM
Sorting files by time / date. TheRealDeal Linux - General 2 08-06-2006 09:35 PM
delete files date wise hardeep_ubhi Linux - General 1 11-16-2004 02:07 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:38 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration