[SOLVED] Strange output from awk

lamont0 · 04-04-2016, 05:49 PM

I execute a script where I initialize a variable and it effects the output. The script does a recursive search thru a directory list all files less than 32 bytes. This works. I then wanted to add a counter to sum up all the bytes. When I initialize a byte counter to "0" a directory is allowed to be processed as output data. If I set the byte counter to "1", the output is as expected. The counter is off by 1 byte and the directory is not listed.

Code:

  ls -lR /home  2> /dev/null | awk '
  BEGIN{print "LIST OF FILES WITH LESS THEN 32 BYTES", bytes=0}
    NF == 9 && $1 !~ /^d/ && $5 < 32, bytes=bytes+$5
  END{print "Total " bytes " bytes"}' > ~/Chapter7/source/awk_data4

Code:

-rw-rw-r-- 1 linux1 linux1    0 Feb 20 20:01 accounta
-rw-rw-r-- 1 linux1 linux1    0 Feb 20 20:01 accountb
-rw-rw-r-- 1 linux1 linux1    0 Feb 20 20:01 accountc
drwxrwxr-x 2 linux1 linux1 4096 Jan 29 13:37 Chapter1

Also, when I initialize the variable it is printed whit the output. How do you suppress that behavior?

Code:

LIST OF FILES WITH LESS THEN 32 BYTES 0

lamont0 · 04-04-2016, 06:11 PM

I resolved the output of the initialized variable by moving the byte counter after the awk invocation: -v bytes=0. The other behavior has not changed.

grail · 04-04-2016, 06:44 PM

You need to go and look at the User guide site for awk as you have several issues in your code.

Specifically:

1. What does a comma do to a print statement

Code:

print "LIST OF FILES WITH LESS THEN 32 BYTES", bytes=0

2. What is the default action if one is not defined

Code:

NF == 9 && $1 !~ /^d/ && $5 < 32, bytes=bytes+$5

3. What happens when there is no delimiter between information and a keyword

Code:

bytes=bytes+$5
  END

4. What does a comma do to a set of tests (I would be interested to know as well as it never occurred to me to put one in??)

Code:

NF == 9 && $1 !~ /^d/ && $5 < 32, bytes=bytes+$5

lamont0 · 04-04-2016, 07:22 PM

Thank you for your reply,
1 - Without the comma I received a syntax error. However, this is not a problem now.
I moved this after "awk" and before the BEGIN.

Quote:

ls -lR /home 2> /dev/null | awk -v counter=0 '

2 - If the bytes counter is not initialized I still get the one directory listed in
the output.
3- If I remove everything after the END then I receive a syntax error.

Quote:

awk: cmd. line:4: END blocks must have an action part

4 - If I remove the "," I receive a syntax error.

Quote:

awk: cmd. line:2: NF == 9 && $1 !~ /^d/ && $5 < 32 counter=counter+$5
awk: cmd. line:2: ^ syntax error

grail · 04-04-2016, 09:30 PM

Hence you need to go and look at more examples. Specifically you need to look up what character to use to delimit one section from another ... I can tell you it is not a comma

mikenash · 04-05-2016, 08:26 AM

Thanks grail! I contemplated about your directions and a light bulb went off.
This is what I finally changed.

Quote:

ls -lR /home 2> /dev/null | awk -v counter=0 '
BEGIN{print "LIST OF FILES WITH LESS THEN 32 BYTES"}
NF == 9 && $1 !~ /^d/ && $5 < 32 && counter=counter+$5
END{print "Total " counter " bytes"}' > ~/Chapter7/source/awk_data4

grail · 04-05-2016, 10:43 AM

Not bad, although you are going to run into problems

Code:

NF == 9 && $1 !~ /^d/ && $5 < 32 && counter=counter+$5

The above reads, if there are 9 fields in the line and the first character of the first field is not 'd' and the fifth field is less than 32. . . so far so good. Now lets assume the first
file(s) you find are all of 0 bytes in length. As the counter is set to 0 initially, each of these will make the total for counter equal to 0, which is regarded as false, hence none of
these files will be printed and yet they all should be ... ie. this is not what you wanted.

I am also still surprised that having no delimiter between $5 and END doesn't cause an issue, but I do believe it then does not show the correct results.
Try creating a small subset in a single directory and test your results.

mikenash · 04-05-2016, 02:33 PM

Hello grail, thank you for catching the omissions. I am still thinking about that one. I understand why it happens but have not though of the solution. Well, I could cheat! Anyway, I am curious about your comment about the delimiter. If I remove the counter then output is correct. Are you referring to parentheses; ()?

grail · 04-05-2016, 04:13 PM

Hmmm ... turns out I have not read the doco in a while either

Previously you needed a semi-colon between items not delimited by curly brackets. This would still be required
if you placed all of the command on a single line, but not with the new lines in the places you have them. The semi-colon would have been your solution though to your first print issue
and it would have told the print command to stop and then setting your variable would not have been included in the output.

You will need to incorporate curly braces into the main part of your script so as to not worry about the zero values causing incorrect output.

mikenash · 04-05-2016, 04:44 PM

Hello grail, thank you for responding. I apologize but I am not quite understanding the explanation. When you state all the command are you stating all 4 lines on only 1 line? Then a semi-colon before the increment of the counter? Also, I have tried curly brace everywhere but I always get syntax errors.

grail · 04-05-2016, 05:04 PM

Code:

NF == 9 && $1 !~ /^d/ && $5 < 32{counter=counter+$5;print}

lamont0 · 04-05-2016, 06:36 PM

Well, I had tried this and the listing for the files was suppressed. Then I thought, if one line provides the file listing and the other the counter then why not use both! That is what I did. Thanks grail, I hope this is correct!

Quote:

NF == 9 && $1 !~ /^d/ && $5 < 32
NF == 9 && $1 !~ /^d/ && $5 < 32{counter=counter+$5}

jpollard · 04-05-2016, 06:45 PM

Or you could include a "print $0;" in the block containing counter=counter=$5.

grail · 04-05-2016, 08:18 PM

Quote:

I had tried this and the listing for the files was suppressed.

Not sure I follow ... you had tried what I wrote and you were getting no output?? The print statement should print the current line

mikenash · 04-06-2016, 09:55 AM

Hello grail, once again thank you. I had tried the curly braces around the counter and I would not get any file listings. If I left the curly braces off then I would get the file listing but lose the zero byte file listings until the counter became positive. The print $0 resolved these two issues. I never saw or used awk before this past Sunday so this was a great learning exercise. Thank you.