[SOLVED] Grep: Return a user message when no hits

kristo5747 · 06-13-2011, 06:22 PM

I have ASCII files to parse that 48 hours old or more ; I can identify them like so

Code:

    find . -name "FILE*TXT"  -mtime +1 -exec ls -ltas '{}' ';'

Some files have a list of hardware errors (we test electronic components), some have none. If the file name has no errors, I still want to display a message like so

Code:

        grep ^err R*VER && echo "No error"
    FILEA.TXT:err	->USB	3910	err	
    FILED.TXT:err No Error

This grep statement works but it seemingly overrides the find() statement above if I run both at the same time... How can I combine the two statements to create a report that lists the filename and error(s) like so

Code:

    FILEA.TXT Button	3320	err
    FILEB.TXT USB	3235	err
    FILEC.TXT IR Remote	2436	err
    FILED.TXT No error

Is it possible to return "No error" with the file name without error? Thanks in advance for your help.

neonsignal · 06-13-2011, 07:09 PM

You may be better off writing a script, but you can do something like this by starting a shell command from the find so that you can run the grep each time:

Code:

find . -name "FILE*TXT" -mtime +1 -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \;

evo2 · 06-13-2011, 07:18 PM

Hi,

not sure I really understand the problem, but I suspect the following might do what you want:

Code:

find . -name 'FILE*TXT' -mtime +1 -print0 | xargs -0 sh -c 'grep -H ^err $@ && echo "$@: No error"'

One problem I have is that you have tow different shell globs for the file name: FILE*TXT and R*VER.

Also you seem to be searching for lines beginning with "err" (grep ^err), yet the output indicates that "err" is at the end of the line.

I think if I understood better what the input files looked like I could give a better solution.

Evo2.

kristo5747 · 06-14-2011, 11:48 AM

Quote:

Originally Posted by neonsignal

You may be better off writing a script, but you can do something like this by starting a shell command from the find so that you can run the grep each time:

Code:

find . -name "FILE*TXT" -mtime +1 -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \;

Beautiful!! It accomplished exactly what I was looking for. The challenge for me was to run grep each time: I could not run figure out a way.

My plan is put the code into a script so the data can be loaded into a mySQL database table. Then, I am going to use the fulltext feature to create a search engine (form based) in PHP.

Thanks again.

kristo5747 · 06-14-2011, 06:15 PM

After running the command

Code:

find . -name "R*VER" -mtime +1 -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \;

my files look like this

Code:

RRR1~COS~COSMETICS~40048~jgmdtv113~1~P~R22-200~029053662549~20110607~102151.VER No error
RRR1~COS~COSMETICS~ETT03~jgm14652.~3~F~R16-500~000907009757~20110607~085109.VER err 3922 	
...

Using the tilde (~) symbol as delimiter, can I extract the fields I want so I get an output like this

Code:

RRR1~COS~COSMETICS~40048~jgmdtv113~1~P~R22-200~029053662549~20110607~102151.VER No error RRR1 COS P
RRR1~COS~COSMETICS~ETT03~jgm14652.~3~F~R16-500~000907009757~20110607~085109.VER err 3922 RRR1 COS F 	
...

I tried the following

Code:

find . -name "R*VER" -mtime +1 -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \;|awk -F~ '{print $0}{print $1"\t"$2"\t"$7"\t"$8"\t"$9"\t"$10}'

but it does not work; Instead, it produces this output (not all fields shown here...)

Code:

RRR1~COS~COSMETICS~40048~jgmdtv113~1~P~R22-200~029053662549~20110607~102151.VER No error
RRR1~COS~COSMETICS~40048~jgmdtv113~1~P~R22-200~029053662549~20110607~102151.VER RRR1 COS P
RRR1~COS~COSMETICS~ETT03~jgm14652.~3~F~R16-500~000907009757~20110607~085109.VER err 3922
RRR1~COS~COSMETICS~ETT03~jgm14652.~3~F~R16-500~000907009757~20110607~085109.VER RRR1 COS F

I'd like to do this is one pass so I can generate a SQL script with INSERT statements...Can it be done?

grail · 06-14-2011, 08:26 PM

Quote:

my files look like this

Code:
RRR1~COS~COSMETICS~40048~jgmdtv113~1~P~R22-200~029053662549~20110607~102151.VER No error
RRR1~COS~COSMETICS~ETT03~jgm14652.~3~F~R16-500~000907009757~20110607~085109.VER err 3922
...

They only look like this after the find and grep. I think what evo2 was asking is what does the inside of each
of the files listed above look like?

I would add that if you are going down the road of using awk, why not simply employ it for the whole task,
ie get rid of grep as it is superfluous when awk can do its task as well. Again to help you do this we would
need to see some of the file contents (can be made up if sensitive, but must still include the main items to be searched for).

kristo5747 · 06-15-2011, 06:19 PM

I came up with this (with help)

Code:

for myfile in `find . -name "R*VER" -mtime +1`
do
     somestr=`grep -H ^err $myfile || echo -e "$myfile\tNo error"`
     echo "$somestr" |sed 's:./::;s:tmp/::;s/:/\t/;'
     echo $somestr | awk -F~ '{print $1"\t"$2"\t"$7"\t"$8"\t"$9"\t"$10}' | sed 's:./::;s:tmp/::;'
done

It is more concise but still outputs on two lines i.e.

Code:

RRR1~COS~COSMETICS~99537~jgmdtv132~1~P~R22-200~029053252648~20110607~094718.VER    No error
RRR1    COS    P    R22-200    029053252648    20110607
RRR1~COS~COSMETICS~ETT03~jgm14652.~1~F~R16-300~000894980523~20110607~084053.VER    err    ->IR Remote Key 1    3310    err   
RRR1    COS    F    R16-300    000894980523    20110607

I am looking to generate dynamically a SQL script with insert statements like so:

Code:

INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8)
VALUES ('RRR1~COS~COSMETICS~99537~jgmdtv132~1~P~R22-200~029053252648~20110607~094718.VER','No error','RRR1','COS','P','R22-200','029053252648','20110607');
INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8)
VALUES  ('RRR1~COS~COSMETICS~ETT03~jgm14652.~1~F~R16-300~000894980523~20110607~084053.VER',' err    ->IR Remote Key 1    3310    err','RRR1','COS','P','R16-300','000894980523','20110607');

All I would need to do is wrap both lines but I am not clear how to do it (still learning awk/sed). Is it possible?

Thanks for your help.

chrism01 · 06-15-2011, 06:58 PM

TBH, I'd write a program (in my case using Perl, but awk would work) to do the whole thing ie right from analysing the orig log files to creating the INSERT statements file, all in one go/program.
It's got to be easier/more flexible/more robust in the long run than the current approach?

kristo5747 · 06-15-2011, 06:59 PM

Quote:

Originally Posted by evo2

Hi,
...
I think if I understood better what the input files looked like I could give a better solution.

Evo2.

I am not permitted by my boss (re: "trade secret") to show you a complete example of a (DOS) file but the section where the error is looks like this in case of an error

Code:

hdr     DTV Error Code List             hdr     ^M
err     ->BIST Login Fail       3922    err     ^M
64      Clean Slot Global Data  00:00:00        Passed  20101110        0^M

and like this in case of no error

Code:

hdr     DTV Error Code List             hdr     ^M
64      Clean Slot Global Data  00:00:00        Passed  0       0^M

No fixed layout, as you can see.

grail · 06-16-2011, 12:24 AM

First I would have to agree with Chris that the current method does not seem optimal.

Quote:

Is it possible?

Of course

Like my previous post where grep was not required, your current example does not need sed.

Quote:

I am not permitted by my boss (re: "trade secret") to show you a complete example

The actual data can be useless information, ie 111 222 aaa ... whatever. What we require is to know what IS important,
ie like:

1. the err on the line - Is it always at the start of the line?
2. What separates one record from the next? ie a blank line a word a symbol??
3. You seem to show - file_name (no)error {if error then show it} and then pieces of the file name. Does this now mean that
there will be only one error per file?
4. Have you taken into account that the DOS based file with ^M at the end of lines will throw you bogus information from
awk and sed unless correctly looked for?

I am sure there will be other things, but these are the few that appear to me based on what is presented so far.

kristo5747 · 06-16-2011, 06:41 PM

With the help, this is what I got

Code:

find . -name "R*VER" -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \; |
        awk -F: '
                BEGIN { q="\047" }
                {
                        file=$1
                        result=$2
                        sub(/^.*\//,"",file)
                        split(file,a,/~/)
                        print "INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8) VALUES (" q file q "," q result q "," q a[1] q "," q a[2] q "," q a[7] q "," q a[8] q "," q a[9] q "," q a[10] q ")"
                }
        '

It works great. Thanks for taking the time. Thread marked as SOLVED.

grail · 06-16-2011, 08:54 PM

Great to see you have a solution

I thought as I keep going on about your seds and greps not being required that it would now be rude not to demonstrate,
of course you may need to test it a little bit as I do not have the input files as you do:

Code:

#!/usr/bin/awk -f

BEGIN { OFS = "\047,\047" 
		insert = "INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8) VALUES (\047"
		noerr = "No error"
		}

file && file != FILENAME{ print insert file,err,a[1],a[2],a[7],a[8],a[9],a[10]"\047)" }

FNR == 1{
	file = FILENAME
	err = noerr
	split(file,a,/~/)
	}

/^err/{
	err = $0
	nextfile
}

END{ print insert file,err,a[1],a[2],a[7],a[8],a[9],a[10]"\047)" }

Please this in a file and make it executable and then run it like so:

Code:

./script.awk R*VER

Now I guessed that there are no directories with this name and your find is only targeting the current directory (even
though you have not set the maxdepth)

If there truly are files in directories below the current one, this can easily be added (let me know)

Also if these are windows files being read we may need to look at the record endings, but your grep makes no allowance so I have not either.