gawk/awk/tcsh - how to convert large number to human readable (Mb,Gb, etc)?

BrianK · 11-07-2007, 08:28 PM

in a script, I run through a file system finding certain files and accumulating their size.

At the end of the script, it generates a report, but the accumulated sizes are HUGE.

I'm woefully ignorant when it comes to [g]awk. Is there a simple way to convert the large number into something more human redable?

angrybanana · 11-07-2007, 09:48 PM

example code:

Code:

awk 'BEGIN{sum=456456;
hum[1024**3]="Gb";hum[1024**2]="Mb";hum[1024]="Kb"; 
for (x=1024**3; x>=1024; x/=1024){ 
        if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x];break }
}}'

output:

Code:

445.76 Kb

bigearsbilly · 11-08-2007, 04:28 AM

Code:

du -sh *
94M     HOME.tgz
32K     noughts.zip
146M    solaris_home_20070814.tgz

linux only

BrianK · 11-08-2007, 12:29 PM

Quote:

Originally Posted by angrybanana

example code:

Code:

awk 'BEGIN{sum=456456;
hum[1024**3]="Gb";hum[1024**2]="Mb";hum[1024]="Kb"; 
for (x=1024**3; x>=1024; x/=1024){ 
        if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x];break }
}}'

output:

Code:

445.76 Kb

Factastic, thanks!

I had to remove the "BEGIN" in order to be able to access data passed in through $1, otherise, this works great.

Thanks again.

angrybanana · 11-08-2007, 01:22 PM

Quote:

Originally Posted by BrianK

Factastic, thanks!

I had to remove the "BEGIN" in order to be able to access data passed in through $1, otherise, this works great.

Thanks again.

Oh yea, used "BEGIN" for demonstration sake (no input). You'll probably use it in "END". Also, I forgot to add in 'bytes', but that should be trivial.

Ronald · 08-13-2010, 09:23 AM

Quote:

Originally Posted by angrybanana

example code:

Code:

awk 'BEGIN{sum=456456;
hum[1024**3]="Gb";hum[1024**2]="Mb";hum[1024]="Kb"; 
for (x=1024**3; x>=1024; x/=1024){ 
        if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x];break }
}}'

output:

Code:

445.76 Kb

This is great !!!

I do have one question

My source is a find and the total size of all found files is greater than 1 Gb. This is ofcourse no problem for this code.. But here is the thing. When the total size is above 1Mb the result is given in Mb's as wel as Kb's . Even when the total size is above 1Gb the results are in Gb's Mb's and Kb's

Any idea how to filter this so that it will only display the correct one?

Ronald · 08-18-2010, 04:53 AM

ok no one with any suggestion(s)

druuna · 08-18-2010, 05:34 AM

Hi,

Could you post an example of what you actually tried to do? The above awk code shows 1 format, be it Gb, Mb or Kb. It does not show Mb and Kb or Gb, Mb and Kb.

One more thing that could be of interest: You posted this from a machine running OS-X. Assuming that the code you tried is also run on that machine: What flavour/version of awk are you using? (awk --version should show both).

Hope this helps.

Ronald · 08-18-2010, 06:04 AM

Quote:

Originally Posted by Ronald

ok no one with any suggestion(s)

Hmm Turns out I must have an error in my format..

When I use a big number in your test script it works just fine.. So the script is good

Nevermind me asking

druuna · 08-18-2010, 06:06 AM

Glad to read you got it sorted out

grail · 08-18-2010, 06:34 AM

Code:

#!/usr/bin/awk -f

BEGIN{	x = 456456

        split("B KB MB GB TB PB",type)

	for(i=5;y < 1;i--)
	    y = x / (2**(10*i))

	print y type[i+2]
}

Ronald · 08-18-2010, 06:35 AM

Quote:

Originally Posted by druuna

Glad to read you got it sorted out

Hi,

I do need some help.. I am quite a noob on Bash and awk

Code:

find ./ -name "*.jpg" -exec ls -l {} \; | awk '{for (I=9;I<=NF;I++){ printf "%s",$I }print " :",$5}{ x++ } { size=size+$5 } END { print "\n" x " Files Found""\n\n" } END { hum[1024**3]="Gb"; hum[1024**2]="Mb"; hum[1024]="Kb"; for (x=1024**3; x>=1024; x/=1024) { if (size>=x) print "\n Total Size= " size/x,hum[x]; } }'

This is the script I am working on.. So I put the code for size in Human readable form at the end (with some minor changes). This works but it gives the 3 outputs (Gb, Mb, Kb) When I add the 'break' at the end (see original code) there is no output of the size.

So I do need some help with this. Maybe you can push me in the right direction..

2nd question: I use print is stead of printf cause I want to print some normal text with the output. I can't seem to fix that with printf..

MAny thanks

Edit: it seems it has something to do with total size.. In my test dir I have only 2 .jpg's with a total size of 228850 bytes. When I run the script (with the ';break' at the end, there is no result. When I creat a fake .jpg with a size of 1Gb the output works good. Also when I make a fake file of 25Mb no output is displayed.

Edit2: ok, I am a bit further.. It has something to do with using print in stead of printf..... when I use printf the output works great.. But How can i put some additional text in the output? So it displays Total Size = ......

druuna · 08-18-2010, 07:13 AM

Hi,

Using grail's code posted in post #11:

Code:

#!/bin/bash

find . -name "*.jpg" -exec ls -l --time-style=long-iso {} \; | \
awk 'BEGIN { totalAmount = "0" }
{ totalAmount = totalAmount + $5 }
{ for (I=8 ; I<=NF ; I++ ) { printf "%s",$I } { print " :",$5} }
END { print "-----------------------------------------------------" ;
printf("Total amount of files : %s - total size : ",NR) ; 
split("B KB MB GB TB PB",type)
  for(i=5;y < 1;i--)
    y = totalAmount / (2**(10*i))
    print y type[i+2]
}'

Because the output of ls -l is not always the same when it comes to the date field, I changed it to ls -l --time-style=long-iso, which does give you a consistent output and a fixed starting field (8).

Hope this helps.

grail · 08-18-2010, 07:48 AM

Whilst I like druuna's solution I have been reading a little about the vagaries of ls and why it should not be used, part of which is pointed out by druuna.
So I would submit this slight variation on the same theme:

Code:

#!/bin/bash

find . -name "*.awk" -exec stat {} \; | 
    awk '/Size:/{sum+=$2}
         END{split("B KB MB GB TB PB",type);
             for(i=5;y < 1;i--)
                 y = sum / (2**(10*i));

             print "Total amount of files : "(NR/7)" - total size : "y type[i+2]
         }'

konsolebox · 08-18-2010, 07:50 AM

I think you can easily get the file size with this command:

Code:

find -type f -name '*.jpg' -printf '%s\n'

I also suggest using -iname instead.