LinuxQuestions.org - Add up sum in file in scientific formulation in a shell script

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Add up sum in file in scientific formulation in a shell script (https://www.linuxquestions.org/questions/programming-9/add-up-sum-in-file-in-scientific-formulation-in-a-shell-script-4175463153/)

Add up sum in file in scientific formulation in a shell script

Dear all,

the subject of the thread is my question:

-Is there any easy way to sum up a file where the numbers are stored in scientific formulation?

(e.g.:

5.4654654E+1
5.4654654E+0
5.4654654E-1

)

Quote:

Originally Posted by fd99 (Post 4957225)

Is there any easy way to sum up a file where the numbers are stored in scientific formulation?

(e.g.:

5.4654654E+1
5.4654654E+0
5.4654654E-1

)

With this InFile ...

Code:

5.4654654E+1

5.4654654E+0

5.4654654E-1

... this awk ...

Code:

awk '{sum=sum+$1; printf "%11.7e\n", $1}

  END{print "-------------"; printf "%11.7e\n", sum}' $InFile >$OutFile

... produced this OutFile ...

Code:

5.4654654e+01

5.4654654e+00

5.4654654e-01

-------------

6.0666666e+01

Daniel B. Martin

At
http://code.nasa.gov
is a set of python tools called "dataProductivityToolKit"
http://code.nasa.gov/project/data-productivity-toolkit/
Sourceforge page
http://sourceforge.net/projects/dptoolkit/

from the readme , a list of programs

Code:

 p.bar            Creates bar charts

 p.binit          Assigns data to 2 dimensional bin structure

 p.cat            Rearrages columnar data into key,x,y format

 p.catToTable    Create a table from data in key,x,y format

 p.cdf            Plots the cumulative distribution

 p.cl            An awk-like math utility

 p.color          Makes color scatter plots

 p.cumsum        Computes the cumulative sum of inputs  

 p.datetime      Converts text-based time stamps to seconds from an epoch

 p.dedup          Removes duplicate keys

 p.distribute    Distribute jobs across computers efficiently

 p.exec          Sequentially run commands read from stdin

 p.gps2utc        Convert gps time to utc time

 p.grab          Grab columns from a file with python-like indexing

 p.grabHeader    Extract the commented header from a file

 p.groupStat      Perform statistics over keyed subgroups of input

 p.hist          Plots a histogram 

 p.htmlWrap      Create an html wrapper for images in a directory

 p.interp        Does polynomial interpolation

 p.join          Join two files on specified key columns

 p.link          Link to files based on specified key columns

 p.linspace      Generate a linear spaced sequence of numbers

 p.map            Plot points on a map

 p.medianFilter  Runs data through a median filter

 p.minMax        Find min/max values in specified data column

 p.multiJoin      Join multiple files together based on key

 p.normalize      Normalizes input data

 p.parallel      Run commands in parallel

 p.parallelSSH    Run commands in parallel across several machines

 p.plot          Plot points on a graph

 p.quadAdd        Add all columns from stdin in quadrature

 p.quantiles      Compute quantiles from input data

 p.rand          Generate a sequence of random numbers

 p.rex            Bring python rex to the command line

 p.scat          Make a scatter plot of input data

 p.sed            A sed-like utility with python syntax

 p.shuffle        Randomly shuffle rows of data

 p.smooth        Smooth data

 p.sort          Sort data based on specified keys

 p.split          Split data based on a supplied delimeter  

 p.strip          Remove comments and/or nans from rows

 p.tableFormat    Nicely format input columns in a table format

 p.template      Bring jinja templates to the command line

 p.utc2gps        Convert utc time to gps time

 p.utc2local      Convert utc time to local time given a lon

p.cumsum
will do this

It is some NASA code that comes in handy from time to time

Code:

$ printf '%s\n' 5.4654654E+1 5.4654654E+0 5.4654654E-1 > infile.txt

$ mapfile -t nums <infile.txt



$ ( IFS=+ ; echo "${nums[*]}" | bc )

16.39639647



$ printf '%.7E\n' "$( IFS=+ ; echo "${nums[*]}" | bc )"

1.6396396E+01

bash's mapfile command loads the lines from the file into an array. The '*' index expansion prints the entire array with the first character of IFS between each entry. Since bash doesn't have floating point arithmetic built-in we have to use an external command like bc to to the actual calculation. Finally, doing the operation inside a (..) subshell means we don't have to do anything to restore the original IFS setting afterwards.

In the second version I sent the output through the shell's built-in printf to convert it to exponential notation. With the proper configuration we could certainly get bc to output it like that directly, but I'm not familiar with how to do it, and I'm too lazy right now to look it up. ;)

Hmm, I did

Code:

 echo "5.4654654E1 5.4654654E0 5.4654654E-1" | sed 's/E/*10^/g;s/ /+/g' |bc

60.1201194

So, I tried

Code:

printf '%f\n' 5.4654654E+1 5.4654654E+0 5.4654654E-1

54.654654

5.465465

0.546547

amending from above.

Am I wrong???

Hi.

The numerical utility package num-utils may be useful:

Code:

#!/usr/bin/env bash



# @(#) s1        Demonstrate column sum of numbers, numsum (num-utils).

# See: http://suso.suso.org/programs/num-utils/



# Utility functions: print-as-echo, print-line-with-visual-space, debug.

# export PATH="/usr/local/bin:/usr/bin:/bin"

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }

pl() { pe;pe "-----" ;pe "$*"; }

db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }

db() { : ; }

C=$HOME/bin/context && [ -f $C ] && $C numsum



FILE=${1-data1}



pl " Input data file $FILE:"

cat $FILE



pl " Results, column sum, select column 1:"

/usr/bin/numsum -c -x 1 $FILE



pl " Results, column sum, all columns:"

/usr/bin/numsum -c $FILE



exit 0

producing:

Code:

% ./s1



Environment: LC_ALL = C, LANG = C

(Versions displayed with local utility "version")

OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64

Distribution        : Debian GNU/Linux 5.0.8 (lenny) 

bash GNU bash 3.2.39

numsum (num-utils) 0.5-9



-----

 Input data file data1:

5.4654654E+1        1

5.4654654E+0        2

5.4654654E-1        3



-----

 Results, column sum, select column 1:

60.66666594



-----

 Results, column sum, all columns:

60.66666594 6

The package was in the Debian repository. If not in yours, see the comment at the beginning of the script for the URL. The utilities are written in perl.

Best wishes ... cheers, makyo

O.K., here's another solution using my current favorite language, pure:

Code:

#!/bin/bash

data=(5.4654654E+1

5.4654654E+0

5.4654654E-1

)

# pure wants a comma-delimited vector, so add a comma after each element

# except the last one.

for ((i=0;i<((${#data[*]}-1));++i))

do

 data[i]="${data[i]},"

done

# The "foldl" function applies the 2 argument function in the second argument to the values in the fourth

# argument (a vector) in turn, from the left, using the third argument as the initial first argument, and

# the result as the first argument for every successive argument.

sum=$(echo "foldl (+) 0.0 [${data[*]}];" | pure -q)

echo "Total = ${sum}"

which produces:

Code:

$ bash fd99 

Total = 60.66666594