gawk/awk/tcsh - how to convert large number to human readable (Mb,Gb, etc)?

Ronald · 08-18-2010, 08:01 AM

Quote:

Originally Posted by druuna

Hi,

Using grail's code posted in post #11:

Code:

#!/bin/bash

find . -name "*.jpg" -exec ls -l --time-style=long-iso {} \; | \
awk 'BEGIN { totalAmount = "0" }
{ totalAmount = totalAmount + $5 }
{ for (I=8 ; I<=NF ; I++ ) { printf "%s",$I } { print " :",$5} }
END { print "-----------------------------------------------------" ;
printf("Total amount of files : %s - total size : ",NR) ; 
split("B KB MB GB TB PB",type)
  for(i=5;y < 1;i--)
    y = totalAmount / (2**(10*i))
    print y type[i+2]
}'

Because the output of ls -l is not always the same when it comes to the date field, I changed it to ls -l --time-style=long-iso, which does give you a consistent output and a fixed starting field (8).

Hope this helps.

Hi,

Thanks for the output.
I have mixed it with my current script and now it works just fine. Many thanks.

the date format did not work on my Apple. I will look into that.

Thanks again!

Ronald · 08-18-2010, 08:18 AM

Quote:

Originally Posted by konsolebox

I think you can easily get the file size with this command:

Code:

find -type f -name '*.jpg' -printf '%s\n'

I also suggest using -iname instead.

Yes!! good tip..
Thanks!

druuna · 08-18-2010, 08:20 AM

Hi,

Looking at the OS_X man page for ls: ls -lT might do the trick.

Just checked on my macbookpro: Output seems consistent when using ls -lT. You also need to adjust the I=8 ; I<=NF ; I++ part to I=10 ......

Hope this helps.

grail · 08-18-2010, 10:20 AM

Cheers konsolebox ... always nice to add something to the repertoire

konsolebox · 08-18-2010, 08:46 PM

Thanks no prob.. always having fun here

Ronald · 08-23-2010, 08:05 AM

Ok,

So maybe I should start a new thread.. but since it is still converting to human readable I figured it is still on topic

So the first part is fixed. output works just fine ..

Next I want to see which users holds how many files (of let's say .jpg) and display the total size of those files

The script so far :

Code:

#!/usr/bin/env bash
#

find ./ -iname "*.jpg" -exec ls -l {} \; | awk 'BEGIN {
# Initialize all Arrays
        size = "0";
        u_count[""]=0;
        all_count[""]=0;
}
{
# Assign field names
        sizes=$5
        user=$3

# Count of number of files
        u_count[user]++;
        all_count["* *"]++;

# Count disc space used
        u_size[user]+=sizes;
        all_size["* *"]+=sizes;
}
{
        for (I=9 ; I<=NF ; I++) { printf "%s",$I } print " :",$5}{ x++ } { size=size+$5 }
END { print "\n" x " Files Found""\n" }
END { hum[1024**4]="Tb"; hum[1024**3]="Gb"; hum[1024**2]="Mb"; hum[1024]="Kb";
        for (x=1024**4; x>=1024; x/=1024) { if (size>=x) { { printf "Total Size = ",NR }
        printf "%.2f %s\n\n",size/x,hum[x];break }
        }
# Output
        for (i in u_count) {
                if (i != "") {
                print u_size[i], u_count[i], i;
                                }
                }
        for (i in all_count) {
                if (i != "") {
                print all_size[i], all_count[i], i;
                                }
                }
} '

Ok ok, There are some double outputs.. (that will be fixed later, it is just for references)

The part where I need some help is in the table output. The total size is now displayed in bytes.. And I want that output the same as the total size which is stated above.. (in Gb/Mb/Kb....)

So in words.. I want the 'u_size[i]' & 'all_size[i]' to be printed in Human readable.

Any idea's ?

regards,

Ronald

grail · 08-23-2010, 08:22 AM

Well the first comment would be that either you use druuna's extended options of the ls command or ditch the ls -l altogether as it is inferior
and very prone to error. (check here)

Quote:

So in words.. I want the 'u_size[i]' & 'all_size[i]' to be printed in Human readable.

So what is the issue? Awk has the ability to create functions so just create one that does your calculation and the print the output.

If your not sure how to create a function then look here.

Ronald · 08-23-2010, 09:14 AM

Quote:

Originally Posted by grail

Well the first comment would be that either you use druuna's extended options of the ls command or ditch the ls -l altogether as it is inferior
and very prone to error. (check here)

So what is the issue? Awk has the ability to create functions so just create one that does your calculation and the print the output.

If your not sure how to create a function then look here.

hmm strange, it works now... although I have to fix some tabs..

ScApi · 10-30-2010, 04:20 AM

First of All sorry for my english, I have a question, I'm editing a cgi script that counts data downloaded by users and shows speeds a chived by them, but the data is in bytes for download data/upload data and speeds, I tried serval things to print them as kB or mB but simply outputs are braking after my mods, what schould I point to :

Hers the CODE:

Code:

#!../haserl
Content-type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "xxx">
<html xmlns="xxx">
<head>
<title>Statystyki</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script type="text/javascript" src="../animation.js"></script>
<script type="text/javascript" src="../script.js"></script>
<link rel="stylesheet" type="text/css" href="../style.css" />
<style>
.download { color: #3399ff;font-weight:bold; float:right;}
.upload {color : #ff9933; font-weight:bold; }
.statrow {background-color: #336699; }
.ip{color:#fff;}
</style>

</head>
<body>
<a href="index.cgi" style="float:left">Powrot panelu</a>
<div style="width:1000px;margin: auto">
<table BORDER=5 style="width:850px; border-width=2; border-color:blue;">
<?
   kompy=$(/usr/sbin/ip -o neigh | awk 'NF>4 {print "KOMP", $1, toupper($5)}')

   uzyNVRAM="T"

   #userzy z pliku
   #users=$(cat ./users | awk '{print "USER", toupper($1), toupper($2)}')
   #echo "$users"

   #userzy ze statycznego dhcp
   adres=$( /sbin/ifconfig br0 | grep "inet addr:" | \
            awk '{split($2,tb,":");split(tb[2],ip,".");
                  print sprintf("%s.%s.%s",ip[1],ip[2],ip[3])
                  }
                 ' \
           )
   mapa=$(/bin/nvram get dhcpd_static | \
           awk -v adres="$adres" \
              'BEGIN
               {RS=">"}
               {
        	        split($0, users, "<")
        	        print "MAPA", sprintf("%s.%s", adres, users[2]) , users[3]
                }' | \
           awk 'NF==3{print $1, $2, $3}' \
           )

#   echo "${mapa}"

   (/usr/sbin/iptables -L upload -xvn   | awk 'NR>2 && $2!=0 {print "U", $2, $7}';                                 <<--- get upload data
    /usr/sbin/iptables -L download -xvn | awk 'NR>2 && $2!=0 {print "D", $2, $8}') | \                             <<--- get download data
    awk 'BEGIN{upT="0";downT="0";}
         {
            ips[$3]=1
            if ($1 == "U")
            {
              up[$3] = $2
              upT = addBigNumber(upT, $2);
            }
            if ($1 == "D")
            {
              down[$3] = $2
              downT = addBigNumber(downT, $2);
            }
          }
         END{
              for (i in ips)
              {
                 split (i, ip, ".")
                 print down[i], ip[1], ip[2], ip[3], ip[4], printNR(down[i]), printNR(up[i]), i, printNR(downT), printNR(upT)
              }
          }
          #drukowanie liczb przedzielonych kropkami dla czytelnosci
          function printNR(a,    t, n, b, i)
          {
              split(a,t,"")
              n=length(a)
              b=""
              for (i=n; i>=1; i--)
              {
                 b=sprintf("%s%s", t[i],b)
                 if (((i-n+2) % 3)==0 && i!=1)b=sprintf("%s%s",".",b)
              }
              return b
          }
          #dodawanie dlugich lancuchow jako liczby calkowite (normalnie awk zaokragla)
          #a, b - dwie liczby calkowite
          function addBigNumber(a, b,    i, l, n, t1, t2, r, w, s, a1, b1)
          {
            a1=a ""
            b1=b ""
            if (length(a1)>length(b1))
            {
              l=length(a1);
              for (i=length(b1)+1; i<=l; i++)
              {
                b1=sprintf("0%s", b1);
              }
            }
            else
            {
              l=length(b1);
              for (i=length(a1)+1; i<=l; i++)
              {
                a1=sprintf("0%s", a1);
              }
            }

            r=0;
            s="";
            n=split(a1, t1, "");
            n=split(b1, t2, "");
            for (i=n; i>=1; i--)
            {
              w=t1[i]+t2[i]+r;
              r=w>9 ? 1 : 0;
              w=r ? w-10 : w;
              s=sprintf ("%d%s", w, s);
            }

            if (r>0) s=sprintf ("%d%s", r, s);
            return s
          }   '  | \
   sort -k1rn -k2n -k3n -k4n -k5n | \
   ( echo "${mapa}"; awk '{print $8, $6, $7, $9, $10}' ) | \
   awk 'BEGIN{i=1;print "<tr><th>LP</th><th>USER</th><th>IP</th><th nowrap>Download [b]</th><th nowrap>Download speed [b/s]</th><th nowrap>Upload [b]</th><th nowrap>Upload speed [b/s]</th></tr>"}
           {
           	 if ($1 == "MAPA") mapa[$2] = $3;
           	 else
           	 {
           	 	if ($1 in mapa) komp = mapa[$1]; else komp = "------ NIE DODANY ------"
           	 	if ($2!=0 || $3!=0)
                     {
           	 	  print "<tr class=statrow>", \
                           "<td valign=\"right\" style=\"color:white; text-align:right; \" >", i++, "</td>", \
           	             "<td nowrap class=\"ip\">", komp, "</td>", \
           	             "<td class=\"ip\">", $1, "</td>", \
           	             "<td valign=\"right\" style=\"color:yellow; text-align:right; \">", $2, "</td>", \                                          <<--- printing download data
                           "<td valign=\"right\" style=\"color:yellow; text-align:right; \" ", sprintf("id=\"down_%s\"", $1)," ></td>", \                <<--- download speed
           	             "<td valign=\"right\" style=\"color:yellow; text-align:right; \" >", $3, "</td>", \                                         <<--- printing upload data
                           "<td valign=\"right\" style=\"color:yellow; text-align:right; \" ", sprintf("id=\"up_%s\"", $1)," ></td>", \                  <<--- upload speed
           	           "</tr>"
                     }
           	 }
           }
        END{
           	 	  print "<tr class=statrow>", \
                           "<td colspan=\"3\" valign=\"right\" style=\"color:white; text-align:right; \" >Razem:</td>", \
           	             "<td valign=\"right\" style=\"color:yellow; text-align:right; \">", $4, "</td>", \
                           "<td valign=\"right\" style=\"color:yellow; text-align:right; \" id=\"down_Razem\" ></td>", \
           	             "<td valign=\"right\" style=\"color:yellow; text-align:right; \" >", $5, "</td>", \
                           "<td valign=\"right\" style=\"color:yellow; text-align:right; \" id=\"up_Razem\" ></td>", \
           	           "</tr>"
           }
       '
?>
</table>
<?
    /bin/sh /jffs/speed/trans_speed
    (echo "${mapa}" | awk '{print $2 ";" $3}'; \
     cat /tmp/home/root/speed.out ) | awk  '
                                          NF==1 {
                                                   split($0, mp, ";")
                                                   mapa[mp[1]] = mp[2]
                                                }
                                          NF>1 {print nazwa($1), $1, $2, $3
                                          }
                                          function nazwa(ip)
                                          {
                                             if (ip in mapa)
                                             {
                                                return mapa[ip]
                                             }
                                             else return ip
                                          }
                                           ' | sort -nrk3 -k1 | \
                                          awk '{down+=$3;
                                                up+=$4;
                                                print "<script language=javascript>",
                                                         sprintf("idD=document.getElementById(\"down_%s\");idD.innerHTML=\"%s\";", $2, $3),
                                                         sprintf("idD=document.getElementById(\"up_%s\");idD.innerHTML=\"%s\";", $2, $4),
                                                      "</script>"
                                               }
                                               END{print "<script language=javascript>",
                                                           sprintf("idD=document.getElementById(\"down_%s\");idD.innerHTML=\"%2.2f\";", "Razem", down),
                                                           sprintf("idD=document.getElementById(\"up_%s\");idD.innerHTML=\"%2.2f\";", "Razem", up),
                                                         "</script>"
                                                  }
                                              '

?>
</div>
</body>
</html>

I know that this would be something simple but I'm missing it.