LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-27-2013, 08:58 PM   #1
johnmccarthy
Member
 
Registered: Jul 2010
Posts: 64

Rep: Reputation: 1
Wink script that extracts last 7 days of a log file


Hello all. I'm new to scripting and wondered if I could get a little help. I would like to execute a script that will extract or display the past 7 days of a log file. The log file is /var/log/messages and below is an example of the date format:

May 11 06:01:40 hostname rtvscand: New virus definition file loaded. Version: 130510c.
May 11 06:01:40 hostname rtvscand: Download of virus definition file from LiveUpdate server succeeded.
May 12 06:01:27 hostname rtvscand: New virus definition file loaded. Version: 130511b.
May 12 06:01:27 hostname rtvscand: Download of virus definition file from LiveUpdate server succeeded.

A friend wrote a script but it is not pulling the last 7 days (please see below)

#!/bin/bash

FILE=/var/log/messages

# Get the julian time for this second
N=`date '+%s'`

# Test: March 1st
#N=`date -j 03010000 '+%s'`

# Create a regular expression to match the last 3 days
REGEXP=""
for I in 1 2 3
do
if [ ! -z "$REGEXP" ]; then
REGEXP="$REGEXP|"
fi

# Get the Month/Day for time N
DAY=`date -r $N '+%b %e'`
REGEXP="${REGEXP}${DAY}"

# Subtract one day
N=`expr $N - 86400`
done

# If today were the 10th of March,
# REGEXP should be "(Mar 10|Mar 9|Mar 8)"
grep -E "^($REGEXP)" $FILE
exit 0


Please help, thanks Johnny Mac
 
Old 06-28-2013, 12:59 AM   #2
Iz3k34l
LQ Newbie
 
Registered: Jun 2013
Location: Texas
Distribution: openSuse
Posts: 5

Rep: Reputation: Disabled
Log extraction

here ya go

Code:
#!/bin/bash
#Author: Travis Smith
#Filename:weeklog.sh
#Date Created: 6-27-2013
#Revision: 1.0.0
#Notes: make sure that what ever you name the file
#make sure to 'chmod +x <filename>.sh' so that its executable

#%a returns the abbreviated weekday name
#%A returns the full weekday name
#%d returns the day of the month 
#%Y returns the full 4 digit year
#%y returns the 2 digit year
#adjust to match your required criteria
#for more options
#http://linux.about.com/od/commands/l/blcmdl1_date.htm
#
#Must use the Double qutoes around the date formatting
#to get the use of blank spaces but after the  '+' sign

#set the date for the log name
TODAY=`(date +"%Y-%m-%d")`
#set the number of days back you want to go
for i in 1 2 3 4 5 6 7
do
WEEKOLD=`(date +"%a %b %d"  --date="$i days ago")`

#This echo is used for testing the variable output
#echo "$WEEKOLD"
cat /var/log/messages | grep "$WEEKOLD" >> "$TODAY".txt

done
hope this helps
 
Old 06-28-2013, 02:20 AM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529
Quote:
Originally Posted by Iz3k34l View Post
Code:
cat /var/log/messages | grep "$WEEKOLD" >> "$TODAY".txt
Do use 'grep' on its own here (also search for "the useless use of cat award"), maybe "anchor" grep search terms with "^"? and finally output will be appended to a file stored in the CWD which may cause unexpected results if the file enlarges beyond capacity of the CWD.
 
Old 06-28-2013, 02:25 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529
My take on things (easily modified to add days):
Code:
#!/bin/sh
# set -Cvxe
LANG=C; LC_ALL=C; export LANG LC_ALL

_help() { echo "${progn} -d (divisor: [h]ours or [m]inutes or [s]econds) -[c]count (max divisor back to start at) -f (log file)"; exit 1; }
checkNumeric () { declare -i chkvar; str="$1"; [ $str -eq 0 ] 2>/dev/null && str=1; let "foo=${str}" 2>/dev/null || return 127; }
[ $# -eq 0 ] && _help; while getopts d:c:f:h OPT; do case "${OPT}" in
  d) case "${OPTARG}" in H|h) DIV=h;; M|m) DIV=m;; S|s) DIV=s;; *) _help;; esac ;;  
  c) USERINPUT="${OPTARG}"; checkNumeric "${USERINPUT}" || _help ;;
  f) LOGFILE="${OPTARG}"; [ -f "${LOGFILE}" ] || _help ;;  h) _help ;;
 esac; done
[ -f "${LOGFILE}" ] || _help; for ((MAX=${USERINPUT};MAX>0;MAX--)); do case $DIV in 
   h) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:' --date="$MAX hours ago"); DISPLAYGREPDATE=$(date +'%b %e %H:' --date="$MAX hours ago");;
   m) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:%M' --date="$MAX minutes ago"); DISPLAYGREPDATE=$(date +'%b %e %H:%M' --date="$MAX minutes ago");;
   s) GREPDATE=$(date +'%b[[:blank:]]\{1,2\}%-e[[:blank:]]%H:%M:%S' --date="$MAX seconds ago"); DISPLAYGREPDATE=$(date +'%b %e %H:%M:%S' --date="$MAX seconds ago") ;;
 esac;  RES=$(tac "${LOGFILE}" | grep -c "^${GREPDATE}"); [ $RES -ne 0 ] && echo "${DISPLAYGREPDATE//./ }: ${RES} p/${DIV}"
done

exit 0
 
Old 06-28-2013, 08:56 AM   #5
Iz3k34l
LQ Newbie
 
Registered: Jun 2013
Location: Texas
Distribution: openSuse
Posts: 5

Rep: Reputation: Disabled
Thank you for this prestigious award, and on my very first post, what an honor

Quote:
Originally Posted by unSpawn View Post
also search for "the useless use of cat award"
 
Old 06-30-2013, 06:21 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529
Quote:
Originally Posted by Iz3k34l View Post
Thank you for this prestigious award, and on my very first post, what an honor
The UUOC award has been and can be given to anyone by anyone regardless of time, place or skill.
 
Old 07-01-2013, 08:36 AM   #7
johnmccarthy
Member
 
Registered: Jul 2010
Posts: 64

Original Poster
Rep: Reputation: 1
Iz3k34l

After running your script I get a file that is blank. I ensured that a it ended with .sh and chmod the file to 770. I exectuted it as root. If you run the file do you get the last 7 days of \var\log\messages?
 
Old 07-01-2013, 08:44 AM   #8
johnmccarthy
Member
 
Registered: Jul 2010
Posts: 64

Original Poster
Rep: Reputation: 1
Iz3k34l

Again, not nothing about scripting. I simply ran your orgianl posting and observed the file created with today's date and there is no data in the file. Below your original posting comments were made but I'm not sure what to make of them or how to utilizing them to edit the script.

Sorry for the confusion. Still don't have a solution.

Johnny Mac
 
Old 07-01-2013, 11:01 AM   #9
Iz3k34l
LQ Newbie
 
Registered: Jun 2013
Location: Texas
Distribution: openSuse
Posts: 5

Rep: Reputation: Disabled
Yes i do,
Code:
Jun 29 23:59:10 VCDweb1 systemd-logind[749]: Removed session 15975.
Jun 29 23:59:14 VCDweb1 sshd[10216]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51843 ssh2
Jun 29 23:59:14 VCDweb1 systemd-logind[749]: New session 15976 of user root.
Jun 29 23:59:15 VCDweb1 systemd-logind[749]: Removed session 15976.
Jun 29 23:59:19 VCDweb1 sshd[10302]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51845 ssh2
Jun 29 23:59:19 VCDweb1 systemd-logind[749]: New session 15977 of user root.
Jun 29 23:59:20 VCDweb1 systemd-logind[749]: Removed session 15977.
Jun 29 23:59:24 VCDweb1 sshd[10388]: Accepted keyboard-interactive/pam for root from 192.168.40.12 port 51850 ssh2
Jun 29 23:59:24 VCDweb1 systemd-logind[749]: New session 15978 of user root.
Jun 29 23:59:25 VCDweb1 systemd-logind[749]: Removed session 15978.
-rw-r--r-- 1 root root 2616701 Jul 1 09:52 2013-07-01.txt

your permissions should should look like this
-rwxr-xr-x 1 root root 912 Jun 28 11:29 weeklog.sh*

use the "chmod +x weeklog.sh" command

next make sure if you have changed the directory path that you use the correct path '/' not '\'.

try this "tail -f /var/log/messages" to make sure you have data that is seven days old


Quote:
Originally Posted by johnmccarthy View Post
After running your script I get a file that is blank. I ensured that a it ended with .sh and chmod the file to 770. I exectuted it as root. If you run the file do you get the last 7 days of \var\log\messages?
 
Old 07-02-2013, 09:37 AM   #10
johnmccarthy
Member
 
Registered: Jul 2010
Posts: 64

Original Poster
Rep: Reputation: 1
Smile Fixed - Whew! Resolved

The problem the orginal script is that the date command does not interpet like BSD. The foundation of the script was based on the date command. Below is a perl script that works in linux and provides the current date and 8 previous days.

#!/usr/bin/perl

use POSIX;

my $log = "/tmp/messages"; # Log file name
my $n = 8; # Number of days to go back

# Get the julian time for this second
my $jultm = time();

# Test: March 10th, 2000 @ midnight
#$jultm = POSIX::mktime(0, 0, 0, 10, 3, 100);

# Create a regular expression to match the last 3 days
my $regexp = "";
for(my $I = $n; $I > 0; $I--)
{
if (length($regexp) > 0)
{
$regexp .= "|";
}

# Get the Month/Day for time N
my $day = POSIX::strftime("+%b %e", localtime($jultm));
$regexp .= $day;

# Subtract one day
$jultm -= 86400;
}

# If today were the 10th of March,
# regexp should be "(Mar 10|Mar 9|Mar 8)"
#print "REGEXP: $regexp\n";
my $cmd = "grep -E '^($regexp)' $log";
#print "CMD: $cmd\n";
system $cmd;
exit 0;
 
Old 07-03-2013, 04:07 PM   #11
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
1) Please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques. Thanks.

2) Since this is Linux Questions, we usually assume you're using Linux and its tools (i.e. the gnu coreutils) unless otherwise specified. So always clearly specify what your environment is if it's in any way non-standard.

3) I'd have probably suggested awk myself before learning about #2, since gawk has a good set of built-in time functions for this kind of thing. It could probably still be done with nawk or posix awk, but it would be more complex.

4) I'm starting to wonder if I shouldn't institute a ${Useless} ${Use} ${Of} ${Variable} ${Brackets} award, since I've been seeing them so often recently.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
tar extracts gzip file even without -z flag ajayyadavmca General 3 03-26-2013 12:33 PM
How do I grep my /var/log/secure file for the past 7 days or so many days? johnmccarthy Linux - Newbie 5 01-04-2013 10:43 PM
Shell script to monitor the log file & kill the process if log is not updating. milu_k Programming 5 07-19-2012 09:23 AM
daysold script - how many days old is a file lrirwin Linux - Software 5 07-12-2012 09:03 AM
linux program that extracts .rar file santasballz Linux - Software 2 03-12-2004 02:36 AM


All times are GMT -5. The time now is 06:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration