grep command returns a large value in RH5 U2

cui_jenny001 · 05-07-2009, 06:03 PM

I encountered an issue with grep command, the return value is wrong(not 0 or 1 or 2).

to reproduce it, i have 3 small testing files in /tmp directory:

1. x.sh
2. myparsers.sh
3. tt

x.sh will call myparsers.sh, which processes tt file

1. cat x.sh

Code:

#!/usr/bin/ksh

i=0
while [ 1 ]
do
  echo "Loop: $i"

  # using ksh -x here to show the return value from grep command
  ksh -x /tmp/myparser.sh  > /tmp/parser_out_$i
  [ $? -ne 0 -o ! -s "/tmp/parser_out_$i" ] &&
  {
    echo "fail\n"
    break
  }

  # remove temp files
  if [ $(( i % 50 )) -eq 0 ]; then
    rm -f /tmp/parser_out_*
  fi

  i=$((i+1))

done

2. cat tt

Code:

#DATAFILES
ORCL:ATAVOL1
ORCL:ATAVOL2

3. cat myparser.sh

Code:

#!/bin/ksh

found=0
cat tt |
while read aline
do
  if [ "$found" -eq 0 ]; then
    echo $aline | grep "#DATAFILES"  >/dev/null
    if [ $? -eq 0 ]
    then
      found=1
    fi
  else
    echo $aline
  fi
done
exit 0

all three files are in /tmp directory. run /tmp/x.sh, sooner or later(a few minutes or a dozen minutes) it will fail at the following command:

+ echo '#DATAFILES'
+ grep '#DATAFILES'
+ 1> /dev/null
+ [ 12276 -eq 0 ]
+ read aline
+ [ 0 -eq 0 ]

The return value here from grep is 12276, sometimes are 8224, 11552, or some other large numbers, and this cause my problem to fail, i tried egrep, same error.

but if I remove the ">/dev/null" in myparsers.sh file, I don't see the failure.

my linux is Red Hat Enterprise Linux Server release 5.2 (Tikanga)(64 bit). I don't see the same problem in redhat release 4 or redhat 5(no update).

not sure if anybody reported this bug and what the solution for me here, as i have many >/dev/null statements in my scripts.

really appreciate your help.

anomie · 05-07-2009, 06:10 PM

Your post is difficult to read because you didn't put the code in code tags.

Example:

Code:

echo 'foo' > my-file

Anyway, instead of redirecting stdout from grep to /dev/null, you might want to just give it the -q switch -- that's what it is there for. See the grep(1) manpages for details.

cui_jenny001 · 05-07-2009, 07:13 PM

thanks!

I just added the code tag and testing the -q option, will update if works or not.

ghostdog74 · 05-07-2009, 07:16 PM

why don't do everything inside 1 script?

cui_jenny001 · 05-07-2009, 07:22 PM

those are simplified code from the code I am working on.

cui_jenny001 · 05-07-2009, 07:45 PM

Thanks anomie! seems the -q option is working, the program is still running after 30 minutes, usually it will fail sooner than that.

btw, to reproduce the issue, I combined all three files together following ghostdog74's suggestion:

Code:

#!/usr/bin/ksh

echo "#DATAFILES" >/tmp/tt
echo "ORCL:DATAVOL1" >>/tmp/tt
echo "ORCL:DATAVOL2" >>/tmp/tt

i=0
while [ 1 ]
do
  echo "Loop: $i"

  found=0
  cat /tmp/tt | while read aline
  do
    if [ "$found" -eq 0 ]; then
      echo $aline | grep "#DATAFILES" >/dev/null
      if [ $? -eq 0 ]
      then
        found=1
      fi
    else
      echo $aline >> /tmp/parser_out_$i
    fi
  done

  [ ! -s "/tmp/parser_out_$i" ] &&
  {
    echo "fail! outfile: /tmp/parser_out_$i\n"
    break
  }

  if [ $(( i % 50 )) -eq 0 ]; then
    rm -f /tmp/parser_out_*
  fi

  i=$((i+1))

done

if fails the at the same grep return value error:

+ grep '#DATAFILES'
+ echo '#DATAFILES'
+ 1> /dev/null
+ [ 24564 -eq 0 ]
+ read aline
+ [ 0 -eq 0 ]
+ grep '#DATAFILES'
+ echo ORCL

ATAVOL1

Thank you for your quick response and help!

cui_jenny001 · 05-08-2009, 01:11 PM

I changed to use -q option, it causes other code to break and I was told not to use -q for portability reason, we should use /dev/null, as mentioned in grep man page:

from man page

-q, --quiet, --silent Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was detected. Also see the -s or --no-messages option.

-s, --no-messages Suppress error messages about nonexistent or unreadable files. Portability note: unlike GNU grep, traditional grep did not conform to POSIX.2, because traditional grep lacked a -q option and its -s option behaved like GNU grep's -q option. Shell scripts
intended to be portable to traditional grep should avoid both -q and -s and should redirect output to /dev/null instead.

but since ">/dev/null" causes grep to return a wrong value(intermittent), what else I can do to avoid it or what could be the root cause from my testing code?

anomie · 05-08-2009, 01:31 PM

What other systems will you be running this on?

One option is to use grep -q 'foo' on Linux systems, and use grep 'foo' >/dev/null on e.g. HP-UX systems.

Also, if you're concerned with portability I would suggest using sh instead of ksh.

-------

If you want to get to the root cause of the issue, you need to start eliminating possibilities. Take it out of the while loop and try to reproduce the problem with just grep and a redirection to /dev/null. If you can't reproduce it that way, then that should tell you something.

cui_jenny001 · 05-08-2009, 02:23 PM

the same set of code runs on solaris/aix/linux.

and there is no problem for the same code running on RH5 and RH4U6, the problem only happens on my RH5U2 systems.

the simple code as the following can quickly reproduce the error:

Code:

#!/bin/ksh

while [ 1 ]
do
  echo "DATAFILES" | grep "DATAFILES" >/dev/null
  if [ $? -ne 0 ]
  then
      echo "Fail."
      break
  fi
done

change to use sh instead of ksh or use -q in linux only both require lots of code changes and testing.

anomie · 05-08-2009, 02:42 PM

FWIW, I tested something similar on a CentOS 5.3 system.

Code:

$ while true ; do echo 'DATAFILES' | grep 'DATAFILES' >/dev/null ; echo $? ; sleep 1 ; done
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

I got bored and hit Ctrl+C. Can you reproduce the problem (with enough manual repetition) without using a while loop?

cui_jenny001 · 05-08-2009, 02:49 PM

hard to reproduce it manually, normally it takes about a few minutes, and failed after thousands of times.

anomie · 05-08-2009, 02:52 PM

Quote:

Originally Posted by cui_jenny001

change to use sh instead of ksh or use -q in linux only both require lots of code changes and testing.

A quick example:

Code:

_os=$(uname)

# .... later in program

case "$_os" in

  "Linux") echo 'foo' | grep -q 'foo' ;;

  *      ) echo 'foo' | grep 'foo' >/dev/null ;;

esac

And I agree -- changing production code requires lots of careful testing. It goes with the territory.

Or submit it as a bug to RH. Or... upgrade to 5.3 if possible and see if it persists.

cui_jenny001 · 05-08-2009, 02:57 PM

yeah, I am thinking to upgrade to 5.3

cui_jenny001 · 05-22-2009, 05:00 PM

well, turns out it was caused by a ksh bug

https://rhn.redhat.com/errata/RHBA-2008-0326.html

anomie · 05-22-2009, 05:08 PM

Thanks for the follow up note.