[SOLVED] how to modify certain sequential patterns in bash or c

Nominal Animal · 05-23-2012, 11:52 PM

Well, start with this awk command:

Code:

awk '#
    BEGIN {
        # AIX awk is stupid, so leave the record separator as the default (newline).
        # RS = "[\t\v\f ]*(\r\n|\n\r|\r|\n)"

        # Fields are separated by asterisks.
        FS = "\\*"

        # In output, use Unix newlines and asterisks between fields when using print.
        ORS = "\n"
        OFS = "*"
    }

    (toupper($1) == "ST") {
        # For reference, save this line for the user.
        ST = $0

        # Reset sum and val.
        val = 0
        sum = 0
    }

    # Add BPR02 to val
    (toupper($1) == "BPR") { val += $3 }

    # Add CLP04 to sum
    (toupper($1) == "CLP") { sum += $5 }

    # Subtract PLB04 from sum
    (toupper($1) == "PLB") { sum -= $5 }

    (toupper($1) == "SE") {
        # Output a warning if sum does not match val.
        if (val != sum)
            printf("BPR02 (%g) does not match CAP04-PLB04 (%g) in %s\n", val, sum, ST)

        # Clear ST and reset sum and val, just to be careful.
        ST = ""
        val = 0
        sum = 0
    }' input-file > output-file

Edited: Johnsfine points out below that the third field name is PLB and not BLP which I originally used. This edit fixes that. Thanks for the heads up, johnsfine. This edited version negates PLB and keeps RS unmodified (as AIX awk does not seem to be able to handle regular expression record separators).

Any questions?

johnsfine · 05-24-2012, 06:35 AM

Quote:

Originally Posted by schneidz

PLB*xxxxx886*20121231*L7*-.14

What is the significance of the - in front of .14?

You seem to be manually adding .14 to the total, not subtracting .14; Is that an error or is the - really supposed to be ignored?

Quote:

Originally Posted by Nominal Animal

# Add CLP04 and BLP04 to sum
(toupper($1) == "CLP") { sum += $5 }
(toupper($1) == "BLP") { sum += $5 }

Your BLP is a typo for PLB? Or am I missing something else here?

schneidz · 05-24-2012, 07:24 AM

^ i checked another example and it seems to always be a negative number so we would need to add the opposite of that field.

schneidz · 05-24-2012, 07:35 AM

Quote:

Originally Posted by schneidz

^ i checked another example and it seems to always be a negative number so we would need to add the opposite of that field.

blp seems to be a typo... testing out nominals code now.

nominals code isnt printing anything out for me (even after commenting out the if statement). i am still tweaking it and will advise.

i got it to run. i commented out RS = "[\t\v\f ]*(\r\n|\n\r|\r|\n)". i think that the aix version of awk will only take one character as record separator so in this case it took the first value which was tab.
thanks,

Nominal Animal · 05-24-2012, 12:43 PM

Quote:

Originally Posted by johnsfine

Your BLP is a typo for PLB? Or am I missing something else here?

Good catch, it indeed is a typo. Thanks!

I edited the script to use PLB instead (negated), to clear ST in the SE rule also (so any mismatches between SE and ST will show up without an ST identifier), and to keep RS as the default (since AIX awk cannot handle regular expressions for the record separator). For the example data, it now outputs

Code:

BPR02 (120) does not match CAP04-PLB04 (100) in ST*835*0001

grail · 05-25-2012, 09:56 AM

So is it something like:

Code:

awk 'BEGIN{OFS=FS="*"}/^ST/,/^SE/ && /^(PLB|CLP)/{c+=($5<0?-1:1)*$5}/^SE/{print c;c=0}' file

schneidz · 05-25-2012, 12:17 PM

+ 1 grail... a tweaked version of nominal-animals code is what solved it for me because it was more comprehensive (it took into account the value of bpr02 and would print errors on sets that dont match)

this part confuses me: ($5<0?-1:1)
if the fifth field is less than 0 then multiply by -1 ? (syntax looks crazy to me: the question mark, colon, and the asterisk $5 outside of the parenthesis. also why is there another search for /^SE/ if you already searched for things between /^ST/,/^SE/ at the BEGINning ?)

i'll never understand awk

thanks,

grail · 05-25-2012, 12:32 PM

Code:

/^ST/,/^SE/ && /^(PLB|CLP)/ - in range ^ST to ^SE look for lines that start with PLB or CLP

{c+=($5<0?-1:1)*$5} - perform this when above is true. If $5 is negative, ie < 0, multiply (*) the value by -1, as you mentioned previously all values are to be positive,
                      if false then multiply by 1, ie. return the value of $5

/^SE/{print c;c=0} - when you reach the end of the range, print the current value and reset to 0 in case there is another range

The ?: combo is called a ternary operation. The process is:

Code:

expression?true:false

schneidz · 07-27-2012, 10:28 PM

i am seeing false positives like this:

Code:

bpr02 (38399.2) does not match clp04+plb04 (38399.3) in ST*835*0001
bpr02 (46762.8) does not match clp04+plb04 (46762.8) in ST*835*0002
bpr02 (13310.3) does not match clp04+plb04 (13310.3) in ST*835*0010
bpr02 (4953.29) does not match clp04+plb04 (4953.29) in ST*835*0015
bpr02 (858.13) does not match clp04+plb04 (858.13) in ST*835*0017

so i am still fin tuning. not sure why it is thinking the same number doesnt equal itself ?

code:

Code:

awk -v d=$delim '#
    BEGIN {
        FS=d
        OFS=d
    }

    (toupper($1) == "ST") {
        # For reference, save this line for the user.
        ST = $0

        # Reset sum and val.
        val = 0
        sum = 0
    }

    # Add BPR02 to val
    (toupper($1) == "BPR") { val += $3 }

    # Add CLP04 and BLP04 to sum
    (toupper($1) == "CLP") { sum += $5 }
    (toupper($1) == "PLB") { sum -= $5 }

    (toupper($1) == "SE") {
        # Output a warning if sum does not match val.
        if (val != sum)
            printf("bpr02 (%g) does not match clp04+plb04 (%g) in %s\n", val, sum, ST)

        # Reset sum and val, just to be careful.
        val = 0
        sum = 0
    }' $1

grail · 07-28-2012, 12:03 AM

Is there possibly a round issue somewhere? Have you checked the data being summed to confirm if val and sum should actually be equal?

On a side not your first example really is not equal

firstfire · 07-28-2012, 01:10 AM

Removed as already useless.

schneidz · 07-28-2012, 08:17 AM

yeah... i used a calculator to make sure they are equal (it will take me a while to scrub out the sensitive data so i can share an example. it is likely due to precision in awk but not sure if that parameter is able to be changed ?

schneidz · 07-28-2012, 09:14 AM

i modified the above from

Code:

    # Add BPR02 to val
    (toupper($1) == "BPR") { val += $3 }

    # Add CLP04 and BLP04 to sum
    (toupper($1) == "CLP") { sum += $5 }
    (toupper($1) == "PLB") { sum -= $5 }

to

Code:

    # Add BPR02 to val
    (toupper($1) == "BPR") { val = sprintf("%.2f", val + $3) }

    # Add CLP04 and BLP04 to sum
    (toupper($1) == "CLP") { sum = sprintf("%.2f", sum + $3) }
    (toupper($1) == "PLB") { sum = sprintf("%.2f", sum + $3) }

and it seems to work for now.