awk / sed question

boyd98 · 01-09-2014, 11:19 AM

I have a file that has date ranges listed as such

123 01/09 <data>
124 01/08 <data>
125 01/07 <data>

Where 123 is a random identifying number, and 01/09 is the date format; January 9th.

I need to grep the file and only pull back the last 30 days worth.

so for example: 01/09 back to 12/09

Any help is greatly appreciated.

Racking my brain -

danielbmartin · 01-09-2014, 12:34 PM

Quote:

Originally Posted by boyd98

I have a file that has date ranges listed as such

123 01/09 <data>
124 01/08 <data>
125 01/07 <data>

Where 123 is a random identifying number, and 01/09 is the date format; January 9th.

I need to grep the file and only pull back the last 30 days worth.

so for example: 01/09 back to 12/09

Any help is greatly appreciated.

Racking my brain -

Is there only one line per day? If so, you need to grab the last 30 lines in the file.

If the number of lines per day is variable (and maybe some days have no record) the problem is more complicated. The past 30 days from today (January 9, 2014) goes back into month number 12. Your sample data contains month and day but not year. Maybe your file contains data from the past ten years. In that case it may be difficult to distinguish records from December 2013 from December 2009.

More information, and more detail, is needed.

Daniel B. Martin

boyd98 · 01-09-2014, 12:56 PM

Thanks Daniel,

This is HP-UX and yes the number of lines per day is varied.

Highlevel I was thinking something like:

#!/bin/ksh
tdate=$(date +"%m/%d")
ldate=$(date -d" -30 day" +"%m/%d")

$ awk '{
if ($2 >= $ldate)
grep desired_data > output
}' inputfile

Although 2 big problems with that are that "date -d" doesn't work in hpux and my if statement I don't know how to do a convert on the dates so 01/08 >= 12/08 translates accordingly

So i probably need some perl one liner to translate the last date and then figure out the convert date and time operators so my math works >=

not really sure

i've looked at it a couple ways, but it keeps getting complex

danielbmartin · 01-09-2014, 02:32 PM

Quote:

Originally Posted by boyd98

i've looked at it a couple ways, but it keeps getting complex

The lack of a year in the input file presents difficulty. The short sample input file you provided is in date-sorted order. Is this guaranteed? If so, it may be feasible to "walk back" in sequence, with logic to handle the 12-month year.

Daniel B. Martin

boyd98 · 01-09-2014, 03:38 PM

Basically it's a log file. It is guaranteed to be sorted by date with the most recent entries at the top. It has a fixed number of records then wraps.

The log file could look like this

123 01/09 <data>
124 01/09 <data>
125 01/07 <data>
126 01/05 <data>

and so forth

basically the ultimate goal is to look back 30 days worth (or 2 weeks worth) and say you had this same error this many times.
I could just grep for data, and count it, but that that won't give me a good feel for the frequency of the same error

Not sure if that helps -

Whoever wrote this program to log like this made this difficult to sort. On top of that the program has an option to look at the log file and query data, and time, but not both at the same time. (sucks)

I'm in the position of admin the program and i want to be able to provide this info

Thanks again for your help -

danielbmartin · 01-09-2014, 04:29 PM

With this InFile ...

Code:

123 01/09 <data>
124 01/08 <data>
125 01/07 <data>
772 01/03 <data>
113 12/16 <data>
433 12/15 <data>
213 12/11 <data>
981 12/09 <data>
113 12/09 <data>
114 12/08 <data>
219 12/08 <data>
115 12/07 <data>
103 11/19 <data>
104 11/12 <data>
105 11/04 <data>

... this awk ...

Code:

awk -F " |/"  \
  'BEGIN{getline; print;
           ThisMonth=$2;ThisDay=$3;
           LastMonth=ThisMonth-1;if (LastMonth==0) LastMonth=12}
        {if ($2==ThisMonth||$2==LastMonth&&$3>ThisDay) print}' $InFile >$OutFile

... produced this OutFile ...

Code:

123 01/09 <data>
124 01/08 <data>
125 01/07 <data>
772 01/03 <data>
113 12/16 <data>
433 12/15 <data>
213 12/11 <data>

Daniel B. Martin

boyd98 · 01-09-2014, 06:02 PM

Awesome man, that helps greatly!

Appreciate it

danielbmartin · 01-10-2014, 07:32 AM

Further thoughts on this subject...

The input file might be large and contain records for multiple years. In that case the previously-posted solution would fail. A small change corrects this flaw. It also reduces execution time because it reads only as much of the input file as needed.

With this InFile ...

Code:

123 01/09 <data from 2014>
124 01/08 <data from 2014>
113 12/16 <data from 2013>
213 12/11 <data from 2013>
981 12/09 <data from 2013>
114 12/08 <data from 2013>
104 11/12 <data from 2013>
105 10/04 <data from 2013>
123 01/09 <data from 2013>
124 01/04 <data from 2013>
125 01/03 <data from 2013>
113 12/15 <data from 2012>
113 12/10 <data from 2012>
114 12/08 <data from 2012>
103 11/17 <data from 2012>
104 10/08 <data from 2012>

... this awk ...

Code:

awk -F " |/"  \
  'BEGIN{getline; print;
       ThisMonth=$2;ThisDay=$3;
       LastMonth=ThisMonth-1;
   if (LastMonth==0) LastMonth=12}
  {if ($2==ThisMonth||$2==LastMonth&&$3>ThisDay) print  
     else exit}' $InFile >$OutFile

... produced this OutFile ...

Code:

123 01/09 <data from 2014>
124 01/08 <data from 2014>
113 12/16 <data from 2013>
213 12/11 <data from 2013>

Daniel B. Martin