Add up sum in file in scientific formulation in a shell script

fd99 · 05-23-2013, 08:46 AM

Dear all,

the subject of the thread is my question:

-Is there any easy way to sum up a file where the numbers are stored in scientific formulation?

(e.g.:

5.4654654E+1
5.4654654E+0
5.4654654E-1

)

danielbmartin · 05-23-2013, 09:03 AM

Quote:

Originally Posted by fd99

Is there any easy way to sum up a file where the numbers are stored in scientific formulation?

(e.g.:

5.4654654E+1
5.4654654E+0
5.4654654E-1

)

With this InFile ...

Code:

5.4654654E+1
5.4654654E+0
5.4654654E-1

... this awk ...

Code:

awk '{sum=sum+$1; printf "%11.7e\n", $1}
  END{print "-------------"; printf "%11.7e\n", sum}' $InFile >$OutFile

... produced this OutFile ...

Code:

5.4654654e+01
5.4654654e+00
5.4654654e-01
-------------
6.0666666e+01

Daniel B. Martin

chrism01 · 05-24-2013, 01:13 AM

You can use bc see post #4 http://www.unix.com/shell-programmin...-notation.html

John VV · 05-24-2013, 11:46 AM

At
http://code.nasa.gov
is a set of python tools called "dataProductivityToolKit"
http://code.nasa.gov/project/data-productivity-toolkit/
Sourceforge page
http://sourceforge.net/projects/dptoolkit/

from the readme , a list of programs

Code:

 p.bar            Creates bar charts
 p.binit          Assigns data to 2 dimensional bin structure
 p.cat            Rearrages columnar data into key,x,y format
 p.catToTable     Create a table from data in key,x,y format
 p.cdf            Plots the cumulative distribution
 p.cl             An awk-like math utility
 p.color          Makes color scatter plots
 p.cumsum         Computes the cumulative sum of inputs   
 p.datetime       Converts text-based time stamps to seconds from an epoch
 p.dedup          Removes duplicate keys
 p.distribute     Distribute jobs across computers efficiently
 p.exec           Sequentially run commands read from stdin
 p.gps2utc        Convert gps time to utc time
 p.grab           Grab columns from a file with python-like indexing
 p.grabHeader     Extract the commented header from a file
 p.groupStat      Perform statistics over keyed subgroups of input
 p.hist           Plots a histogram 
 p.htmlWrap       Create an html wrapper for images in a directory
 p.interp         Does polynomial interpolation
 p.join           Join two files on specified key columns
 p.link           Link to files based on specified key columns
 p.linspace       Generate a linear spaced sequence of numbers
 p.map            Plot points on a map
 p.medianFilter   Runs data through a median filter
 p.minMax         Find min/max values in specified data column
 p.multiJoin      Join multiple files together based on key
 p.normalize      Normalizes input data
 p.parallel       Run commands in parallel
 p.parallelSSH    Run commands in parallel across several machines
 p.plot           Plot points on a graph
 p.quadAdd        Add all columns from stdin in quadrature
 p.quantiles      Compute quantiles from input data
 p.rand           Generate a sequence of random numbers
 p.rex            Bring python rex to the command line
 p.scat           Make a scatter plot of input data
 p.sed            A sed-like utility with python syntax
 p.shuffle        Randomly shuffle rows of data
 p.smooth         Smooth data
 p.sort           Sort data based on specified keys
 p.split          Split data based on a supplied delimeter  
 p.strip          Remove comments and/or nans from rows
 p.tableFormat    Nicely format input columns in a table format
 p.template       Bring jinja templates to the command line
 p.utc2gps        Convert utc time to gps time
 p.utc2local      Convert utc time to local time given a lon

p.cumsum
will do this

It is some NASA code that comes in handy from time to time

David the H. · 05-24-2013, 12:47 PM

Code:

$ printf '%s\n' 5.4654654E+1 5.4654654E+0 5.4654654E-1 > infile.txt
$ mapfile -t nums <infile.txt

$ ( IFS=+ ; echo "${nums[*]}" | bc )
16.39639647

$ printf '%.7E\n' "$( IFS=+ ; echo "${nums[*]}" | bc )"
1.6396396E+01

bash's mapfile command loads the lines from the file into an array. The '*' index expansion prints the entire array with the first character of IFS between each entry. Since bash doesn't have floating point arithmetic built-in we have to use an external command like bc to to the actual calculation. Finally, doing the operation inside a (..) subshell means we don't have to do anything to restore the original IFS setting afterwards.

In the second version I sent the output through the shell's built-in printf to convert it to exponential notation. With the proper configuration we could certainly get bc to output it like that directly, but I'm not familiar with how to do it, and I'm too lazy right now to look it up.

chrism01 · 05-27-2013, 01:16 AM

Hmm, I did

Code:

 echo "5.4654654E1 5.4654654E0 5.4654654E-1" | sed 's/E/*10^/g;s/ /+/g' |bc
60.1201194

So, I tried

Code:

printf '%f\n' 5.4654654E+1 5.4654654E+0 5.4654654E-1
54.654654
5.465465
0.546547

amending from above.

Am I wrong???

makyo · 05-27-2013, 11:43 AM

Hi.

The numerical utility package num-utils may be useful:

Code:

#!/usr/bin/env bash

# @(#) s1	Demonstrate column sum of numbers, numsum (num-utils).
# See: http://suso.suso.org/programs/num-utils/

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C numsum

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

pl " Results, column sum, select column 1:"
/usr/bin/numsum -c -x 1 $FILE

pl " Results, column sum, all columns:"
/usr/bin/numsum -c $FILE

exit 0

producing:

Code:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
numsum (num-utils) 0.5-9

-----
 Input data file data1:
5.4654654E+1	1
5.4654654E+0	2
5.4654654E-1	3

-----
 Results, column sum, select column 1:
60.66666594

-----
 Results, column sum, all columns:
60.66666594 6

The package was in the Debian repository. If not in yours, see the comment at the beginning of the script for the URL. The utilities are written in perl.

Best wishes ... cheers, makyo

PTrenholme · 05-27-2013, 12:54 PM

O.K., here's another solution using my current favorite language, pure:

Code:

#!/bin/bash
data=(5.4654654E+1
5.4654654E+0
5.4654654E-1
)
# pure wants a comma-delimited vector, so add a comma after each element
# except the last one.
for ((i=0;i<((${#data[*]}-1));++i))
do
 data[i]="${data[i]},"
done
# The "foldl" function applies the 2 argument function in the second argument to the values in the fourth
# argument (a vector) in turn, from the left, using the third argument as the initial first argument, and
# the result as the first argument for every successive argument.
sum=$(echo "foldl (+) 0.0 [${data[*]}];" | pure -q)
echo "Total = ${sum}"

which produces:

Code:

$ bash fd99 
Total = 60.66666594