bash script for statistics

jeversondg · 04-22-2007, 10:21 AM

He Guys,

I im writing a bash script to read a file and make a statistics from the contents.
The file have severeal lines and each line 100 fields divide by comma and I need to read only the field 32 and 34 from all lines. The problem is length of the fields are varibles

Do you have any idea how to do it?

Matir · 04-22-2007, 11:12 AM

Something like this should work:

Code:

#!/bin/bash

while read LINE
    do FIELDA=`echo $LINE | cut -d, -s -f 32`
    FIELDB=`echo $LINE | cut -d, -s -f 34`
done

makyo · 04-22-2007, 12:21 PM

Hi.

Some parsing is built into bash, so with a little extra work (and perhaps some reading), one might use:

Code:

#!/bin/bash

# @(#) s2       Demonstrate parsing with -- and setting IFS.

F=${1-data1}
echo
echo " Data line:"
head -1 $F

oldifs="$IFS"
IFS=",$IFS"
while read chars
do
    set -- $chars
        fielda=${32}
        fieldb=$34    # WRONG, see posts below
done < $F

IFS="$oldifs"

echo $fielda
echo $fieldb

Which runs against a trivial data file producing:

Code:

% ./s2

 Data line:
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50
32
34

The sample code shows that positional parameters can be referenced with and without the curly braces (at the cost of wrong results) ... cheers, makyo

( edit 1: call attention to error )

Matir · 04-22-2007, 12:40 PM

Quote:

Originally Posted by makyo

The sample code shows that positional parameters can be referenced with and without the curly braces ... cheers, makyo

Position parameters are useful, but it should be noted that, according to the bash manpage:

Quote:

When a positional parameter consisting of more than a single digit is
expanded, it must be enclosed in braces (see EXPANSION below).

Your code is actually using ${3}4, which of course gives 34 when $3=3, as it does in your case.

makyo · 04-22-2007, 03:27 PM

Hi, Matir.

Thanks for looking over the code, and pointing out the error; it always helps to have more than one pair of eyes looking at things.

I changed the data file to highlight the error, and changed the script to call attention to the error:

Code:

#!/bin/bash

# @(#) s2       Demonstrate parsing with -- and setting IFS.

F=${1-data1}
echo
echo " Data line:"
head -1 $F
echo

oldifs="$IFS"
IFS=",$IFS"
while read chars
do
    set -- $chars
        fielda=${32}
        fieldb=$34
        fieldc=${3}4
        fieldd=${34}

        echo $fielda
        echo $fieldb
        echo $fieldc
        echo $fieldd
done < $F

IFS="$oldifs"

exit 0

Which produces:

Code:

% ./s2

 Data line:
1x,2x,3x,4x,5x,6x,7x,8x,9x,10x,11x,12x,13x,14x,15x,16x,17x,18x,19x,20x,21x,22x,23x,24x,25x,26x,27x,28x,29x,30x,31x,32x,33x,34x,35x,36x,37x,38x,39x,40x,41x,42x,43x,44x,45x,46x,47x,48x,49x,50x

32x
3x4
3x4
34x

Note that Matir's note is correct, using the positional parameter without the braces selects the wrong variable, in this case parameter 3 instead of parameter 34 ... cheers, makyo

( edit 1: typo )

ghostdog74 · 04-22-2007, 06:19 PM

Code:

awk -F "," '{print $32 " " $34}' "file"

makyo · 04-22-2007, 07:08 PM

Hi, jeversondg.

Upon reflection, you didn't say much about how you intend to process the data. If it's a small amount and you intend to do the operations in the shell, then Matir's or my method is probably sufficient.

However, if you have a lot of data, if you need to use something other than the shell, or if you will be scaling up the dataset size, then something like ghostdog74's awk suggestion will use fewer resources. You could also use cut directly on the input file in a like fashion ... cheers, makyo

Matir · 04-22-2007, 09:13 PM

Makyo, thanks for posting the updated and continued examples. I'm glad to see a demonstration of it.