LinuxQuestions.org - [SOLVED] Search within a log file within a time Range

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Search within a log file within a time Range (https://www.linuxquestions.org/questions/linux-newbie-8/search-within-a-log-file-within-a-time-range-888053/)

Search within a log file within a time Range

Hi,

I want to search in my apache log, for events which have occurred say between 11:00 AM to 2:00 PM. I have got few scripts/commands but they are not conclusive, some of then are trying to do an exact match(awk) and for some i am just getting the pattern wrong (eGrep)

I am running RHEL.

Apache log eg:
================================================================
10.92.120.129 - - [24/Jun/2011:01:29:07 -0500] "GET /favicon.ico HTTP/1.1" 302
================================================================

now if i want to find out the logs from 1 PM to 2 PM

some example i had was:

cat <apache.log> | grep -E '07\/Apr\/2011:(0[1-2]:[0-5][0-9]|02:00)'

Can someone help on this, for a small script in which i can pass the start time and end time range, and it should give an output of the rows which fall under that time range.

Regards

Code:

#!/bin/bash

d1=$(date -d "20110624 03:00:00" +%s)

d2=$(date -d "20110624 10:00:00" +%s)



while read line

do

  date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')

  date=$(date -d "$date" +%s)

  [[ $date -ge $d1 && $date -le $d2 ]] && echo $line

done < apache.log

The two dates can be passed as arguments. They must be in a valid format for the date command.

still not working..

Thankyou colucix,

I tried the script but i am not able to get any result, attaching the script and the apache log which i am using to test the script.

=============================================================
#!/bin/bash
d1=$(date -d "20110611 04:15:00" +%s)
d2=$(date -d "20110611 04:20:00" +%s)

while read line
do
date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')
date=$(date -d "$date" +%s)
[[ $date -ge $d1 && $date -le $d2 ]] && echo $line
done < /home_dir/a038010/apache.log

=============================================================

Can you please have a look, if like i am doing something wrong.

Regards
Saurav

Dear colucix,

I have finally made it work, it seems i was doing something wrong. It works perfectly, thanks a ton !!!

here is the working script for me, broke the original script and rebuilt it to understand the working... not much of a change though, added few comments

Quote:

#!/bin/bash
#Please enter the time range in the given format, d1 is start time, d2 is end time
d1=$(date -d "20110606 04:17:00" +%s)
d2=$(date -d "20110606 04:18:00" +%s)

while read line
do
date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')
date=$(date -d "$date" +%s)

if [[ $date -ge $d1 && $date -le $d2 ]]
then
# The below line displays the line which falls in between the time/date range given
# use a grep to filter any particular match in the final result
echo $line
fi

done < apache.log

Edit: (forget what I posted here at the beginning before; just discovered my error. :doh:)

I did a bit of rewriting, replacing the awk command with a few parameter expansions, so now it depends on no external commands other than date. I also altered it to accept the parameters as script arguments.

Code:

#!/bin/bash



# Usage:  scriptname [date1] [date2] logname



# dates should be in the same format as the apache log

# although other formats supported by the date command may work

# (but probably won't since the function strips a single colon from it)



cleandate() {



        local n

        # convert the month to numeric form

        case "$1" in

                *Jan* ) n="${1/Jan/01}" ;;

                *Feb* ) n="${1/Feb/02}" ;;

                *Mar* ) n="${1/Mar/03}" ;;

                *Apr* ) n="${1/Apr/04}" ;;

                *May* ) n="${1/May/05}" ;;

                *Jun* ) n="${1/Jun/06}" ;;

                *Jul* ) n="${1/Jul/07}" ;;

                *Aug* ) n="${1/Aug/08}" ;;

                *Sep* ) n="${1/Sep/09}" ;;

                *Oct* ) n="${1/Oct/10}" ;;

                *Nov* ) n="${1/Nov/11}" ;;

                *Dec* ) n="${1/Dec/12}" ;;

        esac



        # remove first colon from string

        n="${n/:/ }"

        #use date to convert to epoch and print

        date -d "$n" +%s



}



# Take the input dates and process them to epoch with the cleandate function

d1=$( cleandate "$1" )

d2=$( cleandate "$2" )



while read line; do



        # Extract the date for each line.

        # First strip off everything up to the first "[".

        # Then remove everything after the first "]".

        # Finally, straighten up the format with the cleandate function

        date="${line#*[}"

        date="${date%%]*}"

        date=$( cleandate "$date" )



        # If the date falls between d1 and d2, print it

        if [[ $date -ge $d1 && $date -le $d2 ]]; then

                echo "$line"

        fi



done < "$3"



exit 0

It could probably be made more robust, such as making it able to handle multiple date formats.

By the way, please use [code][/code] tags around your code, to preserve formatting and to improve readability. Don't use [quote][/quote] tags for code blocks, as they don't preserve formatting. And for goodness sake don't use unbroken lines of "====" or any other character, as they can cause the page width to expand and force side-scrolling. Thanks!

Apologies for a delayed reply, was busy with too many stuff.
Thank~you very much David for the updated Code, all your points taken, this script is very helpful. I can customize it further if required to search other logs with different date formats. Thanks Again