Search Backwards from line number

d1verjim · 10-24-2011, 08:12 AM

Hi all,

This is my first post on here, though I've been tinkering with linux/unix scripts for a while.

I have written a script that searches a file for a string, and then searches backwards for the first date it comes across. THe reason being I need to search across multiple log files on multiple boxes, and find the most recent occurrence of a string. However the date is not on the same line as the string I'm searching for, as I'm searching for something in a multi line XML input.

The script I've written is this:

###########
PREVLINENUM=1
grep -n $1 $2 | while read LINE; do
LINENUM=`echo $LINE | cut -d: -f1`
STR=`echo $LINE | cut -d: -f2`
sed -n $PREVLINENUM','$LINENUM'p' $2 | tac | grep -m1 "^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
echo $STR
PREVLINENUM=$LINENUM
done
#############

and as you can see, it grep's the file with line numbers, and for each match, searches backwards from that line number to the previous occurrence. This works, but I can't help feeling there has to be a better way, as it's really inefficient.

Any ideas on optimising it?

Many thanks,
James.

zootboy · 10-24-2011, 08:39 AM

Have you considered using some sort of XML parser to do it? It would seem like that would be much more efficient than regexes.

grail · 10-24-2011, 09:47 AM

Without an idea of what the data looks like I would agree with zootboy.

d1verjim · 11-10-2011, 08:52 AM

Thanks for your reply guys (and sorry for the delay in responding).

The files I am parsing are not XML, they are application log files, so look like:

2011-11-09 11:00:02.234214 aThreadId DEBUG A line of the log file that contains all sorts of stuff
2011-11-09 11:00:02.234314 aThreadId DEBUG Another line of the log file that contains all sorts of stuff
2011-11-09 11:00:02.234214 anotherThreadId DEBUG A line of the log file that contains all sorts of stuff about a different thread
2011-11-09 11:00:02.234214 aThreadId DEBUG XML Import received: <?xml version="1.0" encoding="UTF-8"?>
<RootNode>
<A node>blah</A>
<Another node>more blah</AnotherNode>
</RootNode>

and so on...

Because the XML in the logfile is multiline, if I want to find an input with "more blah", I then need to search up to find the timestamp from the log line at the start of that XML.

zootboy · 11-10-2011, 09:01 AM

My first thought would be to try and get your application to output the data more directly. If out has any way to separate its logs, that would probably help.

But I'll assume that you can't. In that case, I would try using sed and regexes to split off the entire multi-line chunk. Parse the xml, then pull off the info you wanted.

But this still screams of hack, so I would definitely search for a better way of getting the data in the first place.

bhanuvrat · 11-10-2011, 09:39 AM

Assuming that the aim is to get the task done and not to learn shell scripting,
the task can be accomplished using a small python script like this:

Code:

import datetime
searchFor = "more blah"
while(True):
    try:
        x = raw_input()
        d=x.split()[0]
    except Exception:
        print "Searched string not found"
        break

    try :        
        d = [int(i) for i in d.split('-')]        
        prevdate = datetime.date(d[0],d[1],d[2])        
    except:
        pass
    
    if searchFor in x :
        print "\n found the search key after ", prevdate
        break

redirect the log output as input to this script and it will do your job
what say?

grail · 11-10-2011, 10:44 AM

Why not use tac (reverse of cat) and then when you find what your looking for you can keep going till you find a date and time at the start of the line.

d1verjim · 11-11-2011, 08:45 AM

Thanks guys.

I'll give these a try.

James.