LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-12-2013, 06:50 AM   #1
santoshi_natesan
LQ Newbie
 
Registered: Jun 2013
Location: India
Distribution: linux
Posts: 2

Rep: Reputation: Disabled
Help with shell script


Hi,
need help in shell script especially in reading the log file. Have to grep a "error" from log file and copy to another file in a job which will be scheduled. During the next run of the job it should start looking for the error ignoring the last updated error on the file.

if this is the last updated erorr
2013-04-29 21:44:11,086 [ERROR] com.ct.gd.web.tags.AdSelectTag - Failed to get Ad Content.

then it should start looking from the log file from the time
2013-04-29 21:45

Confused at how to achieve this. Please help
 
Old 06-12-2013, 07:40 AM   #2
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 16

Rep: Reputation: 233Reputation: 233Reputation: 233
You simply should use another file to record the last timestamp or info that the error was took, and use it next time to know where to start reading again. In bash alone or awk, you could (a bit slowly) loop through all the lines everytime you run the script to get the next error.
Code:
#!/bin/bash

shopt -s extglob || {
    echo "Unable to set extended glob."
    exit 1
}

LOGFILE=logfile.log
MARKERFILE=${LOGFILE}.mark
SOMEWHERE=somewhere.file

D='+([[:digit:]])'
DATETIMEPATTERN="${D}${D}${D}${D}-${D}${D}-${D}${D} ${D}${D}:${D}${D}:${D}${D},${D}${D}${D}"
MARKERFOUND=false
NEXTERRORFOUND=false

if read MARK < "$MARKERFILE" && [[ $MARK == $DATETIMEPATTERN ]]; then
    while read LINE; do
        if [[ $LINE == ${MARK}' [ERROR]'* ]]; then
            MARKERFOUND=true
            while read LINE; do
                if [[ $LINE == ${DATETIMEPATTERN}' [ERROR]'* ]]; then
                    NEXTERRORFOUND=true
                    NEWDATETIME=${LINE%%' [ERROR]'*}  ## assuming our lines don't have multiple " [ERROR]"s, which is unlikely.  Else better use awk.
                    break 2
                fi
            done
        fi
    done < "$LOGFILE"
fi

if [[ $MARKERFOUND == false ]]; then
    # Just read the next error from the start.
    # We could place this same/similar could in a function to prevent redundancy.
    while read LINE; do
        if [[ $LINE == ${DATETIMEPATTERN}' [ERROR]'* ]]; then
            NEXTERRORFOUND=true
            NEWDATETIME=${LINE%%' [ERROR]'*}
            break
        fi
    done < "$LOGFILE"
fi

if [[ $NEXTERRORFOUND == true ]]; then
    # Do the changes.
    echo "$LINE" >> "$SOMEWHERE"  ## Using >> to append and not overwrite.
    echo "$NEWDATETIME" > "$MARKERFILE"
fi
We could also exploit use of grep to have lesser code.

Last edited by konsolebox; 06-12-2013 at 07:55 AM. Reason: Didn't work sooner.
 
Old 06-13-2013, 11:22 AM   #3
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
I think you're in luck, in that the time stamp is basically in the proper ISO-8601 format. This means you should be able to just use standard string comparisons on them.

It would help a bit if you explained a bit more about exactly what the script is supposed to do; the way the code is structured can depend on it. But this is my proof of concept version:

Code:
#!/bin/bash

#define your file locations
#the first is the name of the file the timestamp is stored in
lastfile=/path/to/lastfile
logfile=/path/to/logfile
outfile=/path/to/outputfile

#get the last time stamp from the external file.  If no entry, use 0.
read -r lasttime <"$lastfile"
lasttime=${lasttime:-0}

#this is the error string to grep for.
errorstring='my error string'

#extract all lines with matching errors from the logfile and store in an array.
mapfile -t errmessages < <( grep "$errorstring" "$logfile" )

#now loop through the matched errors and print to the $outfile only
#the ones that have a timestamp less than the $lasttime
for line in "${errmessages[@]}"; do
    cur_stamp=${line:0:19}
    if [[ $cur_stamp < $lasttime || $cur_stamp == $lasttime ]]; then
        continue
    fi
    echo "$line"
done >"$outfile"

#save the most recent $lasttime to the $lastfile, overwriting the old one.
echo "$cur_stamp" >"$lastfile"

exit 0
I haven't tested it, but I think it should work. Note that the "${line:0:19}" expansion extracts the first 20 characters from the line, which I feel is safer than the other possible parameter substitutions.

It might possibly be slow-performing on large datasets too, although since it only loops over the lines extracted from grep, it presumably won't have too much work to do.

Still, if performance is a issue it may be better done in a language like perl.

Last edited by David the H.; 06-13-2013 at 11:32 AM. Reason: some code changes and fixes
 
1 members found this post helpful.
Old 06-13-2013, 11:51 AM   #4
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 16

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by David the H. View Post
#now loop through the matched errors and print to the $outfile only
#the ones that have a timestamp less than the $lasttime
I guess your version is more dependent on time rather than the sequence. It would benefit consistency if somehow logs aren't placed synchronously like somehow being buffered before written, but that would also be troublesome if clock gets different.
Quote:
mapfile -t errmessages < <( grep "$errorstring" "$logfile" )
Just a suggestion to make it more compatible with earlier versions:
Code:
if [[ BASH_VERSINFO -ge 4 ]]; then
    mapfile -t errmessages
else
    errmessages=()
    while read -r line; do
        errmessages[${#errmessages[@]}]=$line
    done
fi < <(exec grep "$errorstring" "$logfile" )  # exec to prevent excess summoning of process
Quote:
Code:
[[ $cur_stamp < $lasttime ]]
And for this one, there are versions of bash that doesn't compare well with '<' or '>' if the second argument is not quoted so better:
Code:
[[ $cur_stamp < "$lasttime" ]]

Last edited by konsolebox; 06-13-2013 at 11:27 PM.
 
Old 06-16-2013, 01:18 PM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Quote:
Originally Posted by konsolebox View Post
I guess your version is more dependent on time rather than the sequence. It would benefit consistency if somehow logs aren't placed synchronously like somehow being buffered before written, but that would also be troublesome if clock gets different.
I'm not sure I follow your meaning here. The whole idea of my script is that it's not based directly on time, but text sorting order. As long as the stamps are formatted in some variation of YYYY-MM-DD HH:MM:SS, the older entry should always sort before the newer one.

Are you saying that we can't trust the log's timestamps for some reason?

I notice that I got the direction wrong in my script comment, though.

Quote:
Just a suggestion to make it more compatible with earlier versions:
Yes, this is a good idea. I sometimes overlook the fact that not everyone is using up-to-date software.

Quote:
And for this one, there are versions of bash that doesn't compare well with '<' or '>' if the second argument is not quoted so better:
Code:
[[ $cur_stamp < "$lasttime" ]]
This is another point I'm not sure about. AIUI, there have been some changes in how bash deals with locale-specific collation order, as addressed by the compat32 and compat40 shell options, but I haven't heard anything about quoting being involved. Do you have any details about this?

And would it affect the current test in any case?
 
Old 06-17-2013, 09:29 AM   #6
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 16

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by David the H. View Post
Are you saying that we can't trust the log's timestamps for some reason?
Well that's just the difference of our approach, nothing is really wrong. And yes sometimes we can't trust it like if two programs write two logs at the same time but the first one was written before the newer one since the first one was held on a buffer for some reasons and not written immediately. Also, a possibility could happen if the time suddenly gets different like on a reboot and/or battery failure, and later corrected. If a clock failure or bad setting occurs and a log happens in which the timestamp is older than the one that's currently marked then it won't be included in the processing.

Quote:
This is another point I'm not sure about. AIUI, there have been some changes in how bash deals with locale-specific collation order, as addressed by the compat32 and compat40 shell options, but I haven't heard anything about quoting being involved. Do you have any details about this?

And would it affect the current test in any case?
Not really critical perhaps I'm not sure. The only thing I'm certain is among older versions of bash when I was just creating one of my scripts the results were different and change from time to time. But I was able to fix it after placing the second argument on quotes. Perhaps it's somehow related to a similar handling of the second argument in == or =~. And I don't think it's related to the locale as no setting about it was changed during the fix.

It could be likely that strings inside quotes were handled differently than open strings on the second argument of < or >.

Last edited by konsolebox; 06-17-2013 at 09:34 AM.
 
Old 06-18-2013, 02:39 AM   #7
santoshi_natesan
LQ Newbie
 
Registered: Jun 2013
Location: India
Distribution: linux
Posts: 2

Original Poster
Rep: Reputation: Disabled
Hi Konsolebox and David,

Thanks for the help. but my bash version is 3 so mapfile doesn't work on it. Also there are two type of error am looking as "Server Error" and Error. because "server error" doesn't have time stamp so the timestamp has to be taken from the next line.

posting my code written which is very simple as i am a beginner.


#!/bin/bash
h1=`hostname`
d1=`date`
ABCLOGS=/path to logfile
XYZlog1=/tmp
for fname in $ABCLOGS
do
abc=$(basename $fname)
f=${fname}/ABCLog.log.servererror
var=`tail -1000 $f | grep -A 20 "Server Error" `
echo "$var" >/tmp/abclogcopy.txt
done
for fname1 in $XYZlog1
do
xyz1=$(basename $fname1)
f1=${fname1}/xyzlogcopy.txt
var1=`cat $f1 | grep -c "Server Error" `
echo $var1
if [ "$var1" -ge "5" ] ;
then
echo "$var1"
if [ ! -z "$var" ] ;
then
echo "$var error on $xyz on server $h1 @ $d1" | mailx -s "error on $xyz" <mail-addr>
#break
else
echo "No matching error found on server $h1 @ $d1" >>/tmp/jobrun.txt
fi
else
echo "Count was not found"
fi
done
 
Old 06-19-2013, 01:53 PM   #8
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
I'm fairly sure my general idea could still be made to work.

mapfile is just a convenience feature that can be replaced with a while loop, as konsolebox demonstrated.

As for the second issue, an array may still be a way to go. Just make sure you grab all the important lines of output, then if an error line matches, you can simply grab the +1 entry and process that.

I'm also considering the possibility of using sed, with a range match including the last updated line, as a way to limit the input.

It would help if you could provide a longer example of the log text, and show us exactly what you want to get from it.


As for your posted script, I'm not going to break it down in detail at this time, but here are a few general scripting points that should be addressed in it, at least:

Don't Read Lines With For!
Useless Use Of Cat
$(..) is highly recommended over `..`
Parameter substitution can replace basename
[[..]] should generally be used for string/file tests, and ((..)) for numerical tests


And please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques. Thanks.
 
Old 07-16-2013, 07:41 AM   #9
bonnydeal
LQ Newbie
 
Registered: Feb 2006
Posts: 28

Rep: Reputation: 26
You should start by analyzing your requirement.
IF the requirement is to copy ERROR messages from the logfile to a different file without repetition then you can simply start a shell background process

% nohup tail -f $LOGFILE | grep ERROR > $OUTFILE &
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script for run an shell script on server using ssh bloodstreetboy Linux - Server 5 01-12-2013 04:23 AM
How to pass command line arguments from one shell script to another shell script VijayaRaghavanLakshman Linux - Newbie 5 01-20-2012 10:12 PM
Executing a Shell script with 654 permissions inside another shell script. changusee2k Linux - Newbie 2 06-07-2011 08:58 PM
shell script problem, want to use shell script auto update IP~! singying304 Programming 4 11-29-2005 06:32 PM


All times are GMT -5. The time now is 05:10 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration