perform -log() on all decimal strings within file

snakefact · 12-12-2009, 05:16 PM

The title pretty much says it.

I have a file that contains tab delimited decimals between 0 and 1, many rows X many-columns.

Does anyone know if/how it would be possible to manipulate each value by -log(X). I know I can do this using excel, but my dataset is huge so it would be much easier to do it with one of two unix commands.

Input Example:
0.25 0.75 0.0000001 0.001 0.99
0.0000001 0.001 0.25 0.75 0.99

Output Example:
0.602059991 0.124938737 7 3 0.004364805
7 3 0.602059991 0.124938737 0.004364805

Suggestions welcome.

Thanks-

evo2 · 12-12-2009, 05:28 PM

You asked for a unix command, so how about awk:

Code:

echo "0.25" | awk '{ print -log($1) }'

Cheers,

Evo2.

colucix · 12-12-2009, 05:48 PM

Good suggestion. awk would be my choice, too. Just pay attention to the fact that in awk the log function gives the natural logarithm, whereas the OP asks for the base-10 logarithm, as clearly shown by the posted example. Hence you have to divide the natural logarithm by log(10):

Code:

$ echo "0.25" | awk '{ printf "%11.9f\n",-log($1)/log(10) }'
0.602059991

GrapefruiTgirl · 12-12-2009, 05:57 PM

I like the above suggestions too, but if for whatever reason they don't appeal to the OP, have a look at the bc man page. bc is the "basic calculator" and does log functions too.

evo2 · 12-12-2009, 05:57 PM

Ok, I realize my awk suggestion is not of much use since it uses base e log. Also it does not address reading and writing files like you requested.

So, to make amends here is a little python script that should do what you want. It's not really unix, but, meh.

Code:

#!/usr/bin/env python
import sys
import math
from math import log
for line in sys.stdin:
    for word in line.split():
        try:
            print -math.log10(float(word)),
        except:
            print word,
    print

This should work even if there are things in the file that aren't numbers.

Assuming you call the script logall
Usage:

Code:

logall < infile > outfile

I'm sure perl people could do this in one line.

Cheers,

Evo2.

colucix · 12-12-2009, 06:15 PM

Quote:

Originally Posted by evo2

Ok, I realize my awk suggestion is not of much use since it uses base e log. Also it does not address reading and writing files like you requested.

Not really true...

Code:

$ awk '{for (i=1; i<=NF; i++) sub($i,-log($i)/log(10),$i); print}' file
0.60206 0.124939 7 3 0.00436481
7 3 0.60206 0.124939 0.00436481

Edit: if you want to match exactly the number of digits in the floating point numbers, as shown in the example output, you can change the value of the CONVFMT built-in variable

Code:

$ awk 'BEGIN{CONVFMT="%.9g"}{for (i= 1; i <= NF; i++) sub($i,-log($i)/log(10),$i); print}' file
0.602059991 0.124938737 7 3 0.0043648054
7 3 0.602059991 0.124938737 0.0043648054

snakefact · 12-12-2009, 06:35 PM

I tried the python script and it works great.

The only problem is some of the values returned have too many characters (eg 0.602059991328).

Is there a way to limit the number of characters per value to say 4 after the decimal?

evo2 · 12-12-2009, 10:59 PM

Quote:

Originally Posted by snakefact

I tried the python script and it works great.

Cool.

Quote:

The only problem is some of the values returned have too many characters (eg 0.602059991328).

Is there a way to limit the number of characters per value to say 4 after the decimal?

There sure is. Taking the queue from colucix, just replace the print line with:

Code:

print '%.9g' % -math.log10(float(word)),

EDIT: I just realized there is an unneeded/unusde line in my original script. You can delete the following line.

Code:

from math import log

Cheers,

Evo2.

snakefact · 12-14-2009, 02:42 PM

Quote:

$ awk 'BEGIN{CONVFMT="%.9g"}{for (i= 1; i <= NF; i++) sub($i,-log($i)/log(10),$i); print}' file

Quote:

Edit: if you want to match exactly the number of digits in the floating point numbers, as shown in the example output, you can change the value of the CONVFMT built-in variable

Id like to be able to limit the number of digits after the decimal. The floating point in the above command limits the number of digits after the last 0.

Ex: 0.00000000000009876 can be returned if the CONVFMT="%.4g"

When I'd want 0.0000

Any suggestions?

GrapefruiTgirl · 12-14-2009, 02:56 PM

I'm not familiar with the perl & python, so while there's likely a way to get the math functions to limit the output places, I don't know it. Meanwhile, what you *could* do if you simply want to chop the end off the decimals, is use sed:

Code:

echo 0.00000000000009876 | sed 's/\(.*\.[0-9][0-9][0-9][0-9]\)\(.*\)/\1/'

..just pipe the output of your formula stuff into the sed statement. NOTE: this doesn't round anything or do anything mathematical; it just chops the string down to 4 places-- that's all.

Sasha

colucix · 12-14-2009, 04:44 PM

Quote:

Originally Posted by snakefact

When I'd want 0.0000

Any suggestions?

If you understand what %.9g you can easily find the answer. It is the notation of printf format used in C and inherited by many scripting languages or specific commands. Section 4.5.2 in the current version of the GAWK manual explains it all. Moreover you will find some detailed explanation and caveats about the usage of the CONVFMT internal variable. In the meanwhile you can give a try to CONVFMT=%.4f and see if it suits your requirement.

evo2 · 12-14-2009, 08:10 PM

%6.4f