script that extracts last 7 days of a log file

johnmccarthy · 06-27-2013, 07:58 PM

Hello all. I'm new to scripting and wondered if I could get a little help. I would like to execute a script that will extract or display the past 7 days of a log file. The log file is /var/log/messages and below is an example of the date format:

May 11 06:01:40 hostname rtvscand: New virus definition file loaded. Version: 130510c.
May 11 06:01:40 hostname rtvscand: Download of virus definition file from LiveUpdate server succeeded.
May 12 06:01:27 hostname rtvscand: New virus definition file loaded. Version: 130511b.
May 12 06:01:27 hostname rtvscand: Download of virus definition file from LiveUpdate server succeeded.

A friend wrote a script but it is not pulling the last 7 days (please see below)

#!/bin/bash

FILE=/var/log/messages

# Get the julian time for this second
N=`date '+%s'`

# Test: March 1st
#N=`date -j 03010000 '+%s'`

# Create a regular expression to match the last 3 days
REGEXP=""
for I in 1 2 3
do
if [ ! -z "$REGEXP" ]; then
REGEXP="$REGEXP|"
fi

# Get the Month/Day for time N
DAY=`date -r $N '+%b %e'`
REGEXP="${REGEXP}${DAY}"

# Subtract one day
N=`expr $N - 86400`
done

# If today were the 10th of March,
# REGEXP should be "(Mar 10|Mar 9|Mar 8)"
grep -E "^($REGEXP)" $FILE
exit 0

Please help, thanks Johnny Mac

Iz3k34l · 06-27-2013, 11:59 PM

here ya go

Code:

#!/bin/bash
#Author: Travis Smith
#Filename:weeklog.sh
#Date Created: 6-27-2013
#Revision: 1.0.0
#Notes: make sure that what ever you name the file
#make sure to 'chmod +x <filename>.sh' so that its executable

#%a returns the abbreviated weekday name
#%A returns the full weekday name
#%d returns the day of the month 
#%Y returns the full 4 digit year
#%y returns the 2 digit year
#adjust to match your required criteria
#for more options
#http://linux.about.com/od/commands/l/blcmdl1_date.htm
#
#Must use the Double qutoes around the date formatting
#to get the use of blank spaces but after the  '+' sign

#set the date for the log name
TODAY=`(date +"%Y-%m-%d")`
#set the number of days back you want to go
for i in 1 2 3 4 5 6 7
do
WEEKOLD=`(date +"%a %b %d"  --date="$i days ago")`

#This echo is used for testing the variable output
#echo "$WEEKOLD"
cat /var/log/messages | grep "$WEEKOLD" >> "$TODAY".txt

done

hope this helps

unSpawn · 06-28-2013, 01:20 AM

Quote:

Originally Posted by Iz3k34l

Code:

cat /var/log/messages | grep "$WEEKOLD" >> "$TODAY".txt

Do use 'grep' on its own here (also search for "the useless use of cat award"), maybe "anchor" grep search terms with "^"? and finally output will be appended to a file stored in the CWD which may cause unexpected results if the file enlarges beyond capacity of the CWD.

unSpawn · 06-28-2013, 01:25 AM

My take on things (easily modified to add days):

Code:

#!/bin/sh
# set -Cvxe
LANG=C; LC_ALL=C; export LANG LC_ALL

_help() { echo "${progn} -d (divisor: [h]ours or [m]inutes or [s]econds) -[c]count (max divisor back to start at) -f (log file)"; exit 1; }
checkNumeric () { declare -i chkvar; str="$1"; [ $str -eq 0 ] 2>/dev/null && str=1; let "foo=${str}" 2>/dev/null || return 127; }
[ $# -eq 0 ] && _help; while getopts d:c:f:h OPT; do case "${OPT}" in
  d) case "${OPTARG}" in H|h) DIV=h;; M|m) DIV=m;; S|s) DIV=s;; *) _help;; esac ;;  
  c) USERINPUT="${OPTARG}"; checkNumeric "${USERINPUT}" || _help ;;
  f) LOGFILE="${OPTARG}"; [ -f "${LOGFILE}" ] || _help ;;  h) _help ;;
 esac; done
[ -f "${LOGFILE}" ] || _help; for ((MAX=${USERINPUT};MAX>0;MAX--)); do case $DIV in 
   h) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:' --date="$MAX hours ago"); DISPLAYGREPDATE=$(date +'%b %e %H:' --date="$MAX hours ago");;
   m) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:%M' --date="$MAX minutes ago"); DISPLAYGREPDATE=$(date +'%b %e %H:%M' --date="$MAX minutes ago");;
   s) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:%M:%S' --date="$MAX seconds ago"); DISPLAYGREPDATE=$(date +'%b %e %H:%M:%S' --date="$MAX seconds ago") ;;
 esac;  RES=$(tac "${LOGFILE}" | grep -c "^${GREPDATE}"); [ $RES -ne 0 ] && echo "${DISPLAYGREPDATE//./ }: ${RES} p/${DIV}"
done

exit 0

Iz3k34l · 06-28-2013, 07:56 AM

Thank you for this prestigious award, and on my very first post, what an honor

Quote:

Originally Posted by unSpawn

also search for "the useless use of cat award"

unSpawn · 06-30-2013, 05:21 PM

Quote:

Originally Posted by Iz3k34l

Thank you for this prestigious award, and on my very first post, what an honor

The UUOC award has been and can be given to anyone by anyone regardless of time, place or skill.

johnmccarthy · 07-01-2013, 07:36 AM

After running your script I get a file that is blank. I ensured that a it ended with .sh and chmod the file to 770. I exectuted it as root. If you run the file do you get the last 7 days of \var\log\messages?

johnmccarthy · 07-01-2013, 07:44 AM

Again, not nothing about scripting. I simply ran your orgianl posting and observed the file created with today's date and there is no data in the file. Below your original posting comments were made but I'm not sure what to make of them or how to utilizing them to edit the script.

Sorry for the confusion. Still don't have a solution.

Johnny Mac

Iz3k34l · 07-01-2013, 10:01 AM

Yes i do,

Code:

Jun 29 23:59:10 VCDweb1 systemd-logind[749]: Removed session 15975.
Jun 29 23:59:14 VCDweb1 sshd[10216]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51843 ssh2
Jun 29 23:59:14 VCDweb1 systemd-logind[749]: New session 15976 of user root.
Jun 29 23:59:15 VCDweb1 systemd-logind[749]: Removed session 15976.
Jun 29 23:59:19 VCDweb1 sshd[10302]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51845 ssh2
Jun 29 23:59:19 VCDweb1 systemd-logind[749]: New session 15977 of user root.
Jun 29 23:59:20 VCDweb1 systemd-logind[749]: Removed session 15977.
Jun 29 23:59:24 VCDweb1 sshd[10388]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51850 ssh2
Jun 29 23:59:24 VCDweb1 systemd-logind[749]: New session 15978 of user root.
Jun 29 23:59:25 VCDweb1 systemd-logind[749]: Removed session 15978.

-rw-r--r-- 1 root root 2616701 Jul 1 09:52 2013-07-01.txt

your permissions should should look like this
-rwxr-xr-x 1 root root 912 Jun 28 11:29 weeklog.sh*

use the "chmod +x weeklog.sh" command

next make sure if you have changed the directory path that you use the correct path '/' not '\'.

try this "tail -f /var/log/messages" to make sure you have data that is seven days old

Quote:

Originally Posted by johnmccarthy

After running your script I get a file that is blank. I ensured that a it ended with .sh and chmod the file to 770. I exectuted it as root. If you run the file do you get the last 7 days of \var\log\messages?

johnmccarthy · 07-02-2013, 08:37 AM

The problem the orginal script is that the date command does not interpet like BSD. The foundation of the script was based on the date command. Below is a perl script that works in linux and provides the current date and 8 previous days.

#!/usr/bin/perl

use POSIX;

my $log = "/tmp/messages"; # Log file name
my $n = 8; # Number of days to go back

# Get the julian time for this second
my $jultm = time();

# Test: March 10th, 2000 @ midnight
#$jultm = POSIX::mktime(0, 0, 0, 10, 3, 100);

# Create a regular expression to match the last 3 days
my $regexp = "";
for(my $I = $n; $I > 0; $I--)
{
if (length($regexp) > 0)
{
$regexp .= "|";
}

# Get the Month/Day for time N
my $day = POSIX::strftime("+%b %e", localtime($jultm));
$regexp .= $day;

# Subtract one day
$jultm -= 86400;
}

# If today were the 10th of March,
# regexp should be "(Mar 10|Mar 9|Mar 8)"
#print "REGEXP: $regexp\n";
my $cmd = "grep -E '^($regexp)' $log";
#print "CMD: $cmd\n";
system $cmd;
exit 0;

David the H. · 07-03-2013, 03:07 PM

1) Please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques. Thanks.

2) Since this is Linux Questions, we usually assume you're using Linux and its tools (i.e. the gnu coreutils) unless otherwise specified. So always clearly specify what your environment is if it's in any way non-standard.

3) I'd have probably suggested awk myself before learning about #2, since gawk has a good set of built-in time functions for this kind of thing. It could probably still be done with nawk or posix awk, but it would be more complex.

4) I'm starting to wonder if I shouldn't institute a ${Useless} ${Use} ${Of} ${Variable} ${Brackets} award, since I've been seeing them so often recently.