Need directory listing of file names with ranges that contain dates and times

lcliver · 08-29-2012, 03:20 PM

I have a directory with the following example file names: they are named based on date/time. exampleYYYYMMDDHHMMSSsyslog.log

example20120826184847syslog.log
example20120826184848syslog.log
example20120826185226syslog.log
example20120826193642syslog.log
example20120827153241syslog.log

I need to get a list of files that are in the range of a user entered START_TIME and END_TIME; for example 20120826184848-20120827153241 (the 26th of August @ x time through the 27th of August @ y time) and I can not figure out how to do this. This directory listing I want to cat into a file so at the end I will have a file (syslog.out) that contains the requested file names.

Does this make sense?

Thank you for any help...I have a feeling this is relatively easy but I am stuck!!! so any help would be greatly appreciated!

Thanks!

casualfred · 08-29-2012, 04:20 PM

I think this should probably work..
There's most likely a very simple elegant solution, but I didn't think of one. Just set this script in your directory of dated log files, make it executable, and run it.

Code:

#!/bin/bash

echo "Usage:"
echo "cd /desired/directory"
echo "listdates.sh [start date] [end date]"
echo ""

# put all the .log files in the current directory in a list
ls -1 *.log > filelist

# find the line number of the highest date in the list
TOPDATELNUM=`grep -n "$1" filelist | cut -f1 -d:`

# find the line number of the lowest date in the list
BOTTOMDATELNUM=`grep -n "$2" filelist | cut -f1 -d:`

# output only files between and including those two dates
sed -n -e $TOPDATELNUM,$BOTTOMDATELNUM"p" filelist > syslog.out

# remove our temporary file
rm filelist

echo "syslog.out created"

Good luck!

byannoni · 08-29-2012, 04:38 PM

Here is a nice little awk script:

Code:

#!/bin/bash
ls -1 | awk -v begin=$1 -v end=$2 '
match( $0, /.*([12][0-9][0-9][0-9][01][0-9][0-3][0-9][0-5][0-9][0-5][0-9][0-5][0-9])syslog\.log/, matches )
{
	if( matches[1] >= begin && matches[1] <= end ) { print }
}'

Usage:

Code:

cd dir_with_syslogs
sh name.sh begin_time end_time > output_file

rigor · 08-29-2012, 04:38 PM

casualfred's shell script approach is a clever one. You might need to adjust it a bit in case a file name with the exact date and time the user entered, doesn't exist. If you don't mind including an awk program as part of the solution, and if I understood what you wanted, this appears to work do a comparison, whether or not a file exists with a name containing the exact date/time the User enters.

I put the list of file names you provided in a file named data.txt and the following program in a file named date_list.gawk:

Code:

(  NR  ==  1  )  {  start_date = $1 + 0 ;  end_date = $2 + 0 ; }
# File name pattern: example20120826184847syslog.log
(  NR  >  1  )  {  match( $0 ,  /([0-9]+)/ )  ;  file_date = substr( $0 , RSTART , RLENGTH ) + 0 ; if ( ( file_date >= start_date )  &&  ( file_date <= end_date ) )  print $0 ; }

Then when I run this command sequence, to input a date range and the list of file names to the awk program:

Code:

( echo 20120826184848 20120827153241 ; cat data.txt ) | gawk -f date_list.gawk

I get this output:

Code:

example20120826184848syslog.log
example20120826185226syslog.log
example20120826193642syslog.log
example20120827153241syslog.log

Hope this helps.

chrism01 · 08-29-2012, 05:58 PM

casualfred's soln is a nice shell script one, but for paranoia mode, I'd sort the list

Code:

ls -1 *.log |sort > filelist

# or, if you know the files mtimes haven't been tampered with

ls -1t *.log > filelist

# add -r switch if you want the opposite order

lcliver · 08-30-2012, 07:16 AM

Thank you for the feedback!

I came up with this late last night but it takes a few seconds to run so maybe it is clean on the eye, but not clean running.

for i in `ls *2012082919{5923...5963}* 2>/dev/null`;
do
echo $i >> syslog.out;
done

Is there some reason not to do it this way?

lcliver · 08-30-2012, 07:38 AM

The date_list.gawk works much faster than mine! Thanks!

byannoni · 08-30-2012, 08:32 AM

Here is one that is completely bash:

Code:

#!/bin/bash
for file in *; do
	time=${file:$(expr match $file '[^0-9]*'):14}
	if [[ $time -ge $1 && $time -le $2 ]]; then
		echo $file
	fi
done

Same usage as before:

Code:

cd dir_with_syslogs
sh name.sh begin_time end_time > output_file