LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-24-2011, 03:02 AM   #1
tonan
Member
 
Registered: Nov 2003
Location: india, New Delhi
Distribution: Red Hat Linux
Posts: 62

Rep: Reputation: 16
Unhappy Search within a log file within a time Range


Hi,

I want to search in my apache log, for events which have occurred say between 11:00 AM to 2:00 PM. I have got few scripts/commands but they are not conclusive, some of then are trying to do an exact match(awk) and for some i am just getting the pattern wrong (eGrep)

I am running RHEL.

Apache log eg:
================================================================
10.92.120.129 - - [24/Jun/2011:01:29:07 -0500] "GET /favicon.ico HTTP/1.1" 302
================================================================

now if i want to find out the logs from 1 PM to 2 PM

some example i had was:

cat <apache.log> | grep -E '07\/Apr\/20110[1-2]:[0-5][0-9]|02:00)'


Can someone help on this, for a small script in which i can pass the start time and end time range, and it should give an output of the rows which fall under that time range.

Regards
 
Old 06-24-2011, 03:32 AM   #2
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
Code:
#!/bin/bash
d1=$(date -d "20110624 03:00:00" +%s)
d2=$(date -d "20110624 10:00:00" +%s)

while read line
do
  date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')
  date=$(date -d "$date" +%s)
  [[ $date -ge $d1 && $date -le $d2 ]] && echo $line
done < apache.log
The two dates can be passed as arguments. They must be in a valid format for the date command.
 
Old 06-28-2011, 07:17 AM   #3
tonan
Member
 
Registered: Nov 2003
Location: india, New Delhi
Distribution: Red Hat Linux
Posts: 62

Original Poster
Rep: Reputation: 16
still not working..

Thankyou colucix,

I tried the script but i am not able to get any result, attaching the script and the apache log which i am using to test the script.



=============================================================
#!/bin/bash
d1=$(date -d "20110611 04:15:00" +%s)
d2=$(date -d "20110611 04:20:00" +%s)

while read line
do
date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')
date=$(date -d "$date" +%s)
[[ $date -ge $d1 && $date -le $d2 ]] && echo $line
done < /home_dir/a038010/apache.log

=============================================================


Can you please have a look, if like i am doing something wrong.

Regards
Saurav
Attached Files
File Type: log apache.log (142.6 KB, 16 views)
 
Old 06-28-2011, 09:24 AM   #4
tonan
Member
 
Registered: Nov 2003
Location: india, New Delhi
Distribution: Red Hat Linux
Posts: 62

Original Poster
Rep: Reputation: 16
Thumbs up

Dear colucix,

I have finally made it work, it seems i was doing something wrong. It works perfectly, thanks a ton !!!


here is the working script for me, broke the original script and rebuilt it to understand the working... not much of a change though, added few comments


Quote:
#!/bin/bash
#Please enter the time range in the given format, d1 is start time, d2 is end time
d1=$(date -d "20110606 04:17:00" +%s)
d2=$(date -d "20110606 04:18:00" +%s)

while read line
do
date=$(echo $line | awk 'BEGIN{ FS = "[][]" }{ gsub(/\//," ",$2); sub(/:/," ",$2); sub(/-.*/,"",$2); print $2 }')
date=$(date -d "$date" +%s)

if [[ $date -ge $d1 && $date -le $d2 ]]
then
# The below line displays the line which falls in between the time/date range given
# use a grep to filter any particular match in the final result
echo $line
fi

done < apache.log

Last edited by tonan; 06-28-2011 at 09:25 AM.
 
Old 06-28-2011, 10:25 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Edit: (forget what I posted here at the beginning before; just discovered my error. )

I did a bit of rewriting, replacing the awk command with a few parameter expansions, so now it depends on no external commands other than date. I also altered it to accept the parameters as script arguments.
Code:
#!/bin/bash

# Usage:  scriptname [date1] [date2] logname

# dates should be in the same format as the apache log
# although other formats supported by the date command may work
# (but probably won't since the function strips a single colon from it)

cleandate() {

	local n
	# convert the month to numeric form
	case "$1" in
		*Jan* ) n="${1/Jan/01}" ;;
		*Feb* ) n="${1/Feb/02}" ;;
		*Mar* ) n="${1/Mar/03}" ;;
		*Apr* ) n="${1/Apr/04}" ;;
		*May* ) n="${1/May/05}" ;;
		*Jun* ) n="${1/Jun/06}" ;;
		*Jul* ) n="${1/Jul/07}" ;;
		*Aug* ) n="${1/Aug/08}" ;;
		*Sep* ) n="${1/Sep/09}" ;;
		*Oct* ) n="${1/Oct/10}" ;;
		*Nov* ) n="${1/Nov/11}" ;;
		*Dec* ) n="${1/Dec/12}" ;;
	esac

	# remove first colon from string
	n="${n/:/ }"
	#use date to convert to epoch and print
	date -d "$n" +%s

}

# Take the input dates and process them to epoch with the cleandate function
d1=$( cleandate "$1" )
d2=$( cleandate "$2" )

while read line; do

	# Extract the date for each line.
	# First strip off everything up to the first "[".
	# Then remove everything after the first "]".
	# Finally, straighten up the format with the cleandate function
	date="${line#*[}"
	date="${date%%]*}"
	date=$( cleandate "$date" )

	# If the date falls between d1 and d2, print it
	if [[ $date -ge $d1 && $date -le $d2 ]]; then
		 echo "$line"
	fi

done < "$3"

exit 0
It could probably be made more robust, such as making it able to handle multiple date formats.


By the way, please use [code][/code] tags around your code, to preserve formatting and to improve readability. Don't use [quote][/quote] tags for code blocks, as they don't preserve formatting. And for goodness sake don't use unbroken lines of "====" or any other character, as they can cause the page width to expand and force side-scrolling. Thanks!

Last edited by David the H.; 06-28-2011 at 10:32 AM.
 
1 members found this post helpful.
Old 08-25-2011, 04:10 AM   #6
tonan
Member
 
Registered: Nov 2003
Location: india, New Delhi
Distribution: Red Hat Linux
Posts: 62

Original Poster
Rep: Reputation: 16
Apologies for a delayed reply, was busy with too many stuff.
Thank~you very much David for the updated Code, all your points taken, this script is very helpful. I can customize it further if required to search other logs with different date formats. Thanks Again
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Gnuplot - Combining a linear range and log-scale range in the same X-axis llauro Linux - Software 2 04-26-2011 01:55 PM
[SOLVED] where is time on the squid log file bunpis Linux - Newbie 2 04-13-2011 06:15 PM
Clean Up Log - Search for Pattern in Log file and Output result bridrod Linux - Newbie 10 01-05-2010 10:49 AM
How to search logs between two timestamps in a log file. ram.venkat84 Linux - Newbie 2 11-19-2009 02:04 AM
Why Do I need to run updatedb each time I search for a file or folder? lennysokol Linux - Software 4 05-30-2005 11:49 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:45 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration