[SOLVED] how to make "ls" display the real size for directories

ararus · 05-04-2008, 09:21 AM

Quote:

Originally Posted by forrestt

I don't have any problem with someone conceptualizing a directory as a container. We are humans, and this is how we think, and to know how much stuff is in a container is a useful piece of knowledge (if it weren't we wouldn't have the du command). Thinking of directories as containers makes things easier to visualize and keep track of things in our minds. It may not be the way the file-system actually works, but it is how most people think, and makes perfect sense whether you have hard links or not. To visualize a directory as a database could only be useful if you are someone who is actually designing or programming a file-system. For anyone else, to do so is making things way more complicated than they need to be. If someone knows what the size entry in ls means for a directory entry, then they're fine. But, if they don't, then it needs to be explained, because the natural way of thinking is to assume it means something that it does not. It isn't the conceptualizing that is wrong. It is the assumption that a particular tool is giving one piece of information, when it is actually giving another piece, that is wrong. That is fixed by teaching others how to use our tools.

My 2 pennies,

Forrest

You can conceptualize it any way you want as long as you understand how it really works and that a directory is not really a container. Problems arise when people don't understand this.

I guess this is a bad time to tell you that files don't really have names?

louisJ · 05-05-2008, 09:47 AM

Quote:

Originally Posted by colucix

Following the suggestion from forrestt, you can define a shell function like this, using awk to parse the output of ls -l:

Code:

function lsd () {
ls -l | gawk '
   substr($1,1,1)=="d"{
      ("du -bs " $NF) | getline size
      split(size,size_)
      sub($5,size_[1],$5)
   }
   { printf "%s %2s %s %s %10s %s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,$8,$9}'
}

then every time you type "lsd" a long listing is produced with the size of directories substituted by the total size of their content.

Where do I type it?

Tinkster · 05-05-2008, 01:19 PM

Quote:

Originally Posted by louisJ

Where do I type it?

In your ~/.bashrc
Or /etc/bashrc (bashprofile) if you have more than one user.

Cheers,
Tink

gross6 · 12-26-2010, 07:58 PM

@ colucix
I used your function in the following script:

Code:

#!/bin/bash
#file lsd
#provide disk usage of directories in ls -lh listing

if [ $# -ne 1 ] #if no argument, assign pwd to var a
then
a='./'
else a=$1  #else assign argument to var a
fi

len=${#a} #extract length of a

#count char of a until the last occurrence of "/"
e=`expr match $a '.*/'`

#if count not equal to length of a
#(that is: argument without a final slash)
if [ "$e" -ne "$len" ]
then
a=${a}/  #add slash to argument
fi

#change the 5th field of "ls -lh" with "du -hs" 
ls -lh $a | awk -v dir=$a '
     substr($1,1,1)=="d"{
     ("du -hs " dir$NF) | getline size
     split(size,size_)
     sub($5,size_[1],$5)
     }
     { printf "%s %3s %s %s %6s %s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,$8,$9}'

#print the whole dimension of argument
echo -n "Grand total   "; du -sh $a

The script runs well, but symbolic links and directories having long names and some spaces give problems. The link name is truncated after -> and directories name are truncated too. In the latter case the last word of the name become a directory to scan.
In example:

"cdrom -> /media/cdrom" is printed as "cdrom ->"

and while the command "ls -lh xxxxxxxx/" prints:

"drwxr-xr-x 3 gross6 gross6 4.0K 2009-09-22 12:13 Yyyyyyyyyyyy Www Dddddd zzzzzzzzzz bbb Kkkkkkkkk"

the command "./lsd xxxxxxxx/" prints:

"du: cannot access `xxxxxxxx/Kkkkkkkkk': No such file or directory
drwxr-xr-x 3 gross6 gross6 528K 2009-09-22 12:13 Yyyyyyyyyyyy Www"

The d's, z's, and b's not printed.

Any suggestions and/or improvements to the script?
Thanks

colucix · 12-27-2010, 03:09 AM

To avoid the problem of spaces in the directory names, you have to use an expression in place of $NF or $9 in the awk script. For example:

Code:

substr($0,index($0,$9))

this extract the substring that begins at the position of the 9th field to the end of the record.

Regarding the links, you can try a similar approach by extracting the substring starting three positions after the -> symbol. Example:

Code:

ls -l | awk 'substr($1,1,1) == "l" {print substr($0,index($0,"->")+3)}'

However, I need to rethink the whole approach, since there is something that does not convince me.

gross6 · 12-27-2010, 05:11 AM

Thank you very much colucix. Very kind of you.
I am a little busy right now but I'll try your code as soon as I can.

gross6

gross6 · 12-28-2010, 03:14 PM

I traied this code with some directories having spaces and apostrophe:

Code:

ls -lh  | awk '
	substr($1,1,1)=="d"{
	("du -hs "substr($0,index($0,$8))) | getline size
	split(size,size_)
	sub($5,size_[1],$5)
	}
{ printf "%s %3s %s %s %6s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,substr($0,index($0,$8))}'

It works well with printf but du -hs scan every word of a directory.

The apostrophe gives this error:
/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file

Finally the dimension of following n directories was the same, where n was the number of word of the first directory.

I tried to assign the substr to a variable, example:

Code:

awk -v z=substr($0,index($0,$8)) '
substr($1,1,1)=="d"{
	("du -hs" $z....

But there were no improvements.

By the way, in my system the field of directories is number 8 not 9.

However I continue to try...

Thanks colucix for your concern about the subject.

catkin · 12-28-2010, 09:37 PM

Any reason for not downloading the ls source and modifying that to build a variant or add an option?

gross6 · 12-30-2010, 05:12 AM

Quote:

Originally Posted by catkin

Any reason for not downloading the ls source and modifying that to build a variant or add an option?

Reading of ls.c and du.c of coreutils shows me that is not a simple task to modify such a code.
It needs a deep knowledge of C language and filesystem structure that I have not. Till now...
This is one of the reasons. At least for me.
Bye
gross6

catkin · 12-30-2010, 05:24 AM

Quote:

Originally Posted by gross6

Reading of ls.c and du.c of coreutils shows me that is not a simple task to modify such a code.
It needs a deep knowledge of C language and filesystem structure that I have not. Till now...
This is one of the reasons. At least for me.
Bye
gross6

That's valid

Was just an idea.

gross6 · 01-09-2011, 12:33 AM

This code

Code:

#!/bin/bash
#file lsd
#provide disk usage of directories recursively in ls -lh listing

if [ $# -ne 1 ] #if no argument, assign pwd to var a
then
a='./'
else a=$1  #else assign argument to var a
fi

len=${#a} #extract length of a

#count char of a until the last occurrence of "/"
e=`expr match $a '.*/'`

#if count not equal to length of a
#(that is: argument without a final slash)
if [ "$e" -ne "$len" ]
then
a=${a}/  #add slash to argument
fi

#change the 5th field of "ls -lh" with "du -hs" 
ls -lh $a | awk -v dir=$a '
     substr($1,1,1)=="d"{
     ("du -hs "dir$8"*") | getline size
     split(size,size_)
     sub($5,size_[1],$5)
     }
     substr($1,1,1)=="t"{
     printf ("Files total %s\n",$2)
     }
     substr($1,1,1)!="t"{
     printf ("%s %3s %s %s %6s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,substr($0,index($0,$8)))
     }'

#print the whole dimension of argument
echo -n "Grand total   "; du -sh $a

works well with spaces in dir names.

But there are still troubles with apostrophe in dir name, and when the user and/or group name coincides with dir name.
In summary it mostly works.

Indeed it's only a workaround, not a solution...

Bye
gross6

chrism01 · 01-10-2011, 12:22 AM

With links you're getting into the qn of whether the final file's data (or dir) pointed to should be included or not.
After all, in reality, only the link entry is 'inside' the dir being measured. The actual file is 'inside' the other dir where it really lives...
NB: 'inside' because, as above, a dir file (aka inode) is only an index, it does not physically contain the file's (data) listed, which is why you can fill a partition from any dir ie each dir 'area' does not reserve a space for potential files...

https://secure.wikimedia.org/wikipedia/en/wiki/Inode
https://secure.wikimedia.org/wikiped...nter_structure

piyusharora420 · 01-10-2011, 02:01 AM

i can explain with simple example.
type exactly what i am going to

Quote:

du -h /home

you will see directories with size in kbs

gross6 · 01-10-2011, 11:16 PM

Code:

#!/bin/bash
#file lsd
#provide disk usage of directories recursively in ls -lh listing

if [ $# -ne 1 ] #if no argument, assign pwd to var a
then
a='./'
else a=$1  #else assign argument to var a
fi

len=${#a} #extract length of a

#count char of a until the last occurrence of "/"
e=`expr "$a" : '.*/'`

#if count not equal to length of a
#(that is: argument without a final slash)
if [ "$e" -ne "$len" ]
then
a=${a}/  #add slash to argument
fi

#change the 5th field of "ls -lh" with "du -hs" 
ls -lh "$a" | awk -v dir=$a '
     substr($1,1,1)=="d"{
     ("du -hs "dir$8"*") | getline size
     split(size,size_)
     sub($5,size_[1],$5)
     }
     substr($1,1,1)=="t"{
     printf ("Files total %s\n",$2)
     }
     substr($1,1,1)!="t"{
     printf ("%s %3s %s %s %6s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,substr($0,index($0,$8)))
     }'
  
#print the whole dimension of argument
echo -n "Grand total   "; du -sh "$a"

#modifications 2011/01/11


# {z=(substr($0,index($0,$8),1)"*")} 
# z=`ls -l | awk '{printf "%s:",substr($0,index($0,$8))}'`
# ls -lh | awk '{print substr($0,index($0,$8))}' | sed -e "s/ /\\\ /g"
# ("z=$(`ls -l $a | awk {print substr($0,index($0,$8))}`)") 
# ("du -hs "dir$8"* 2>/dev/null") | getline size

# s=`echo $a | sed -e "s/ /\\ /g" -e "s/'/\'/g"`
# ("du -hs $s"$8"*") | getline size

#printf -v s '%q' ${a}
#ls -lh "$a" | awk -v dir="${s}" '
#     substr($1,1,1)=="d"{
#     ("du -hs "dir$8"*") | getline size

This is my last try... it's a little better than the previous one.
You can see my attempts commented at the bottom of the code but it's difficult for me to find a code that works with spaced dir names listed and spaced name argument of script as well.
Not to mention apostrophes.

Bye :-(
gross6

gross6 · 01-14-2011, 09:22 AM

Code:

#!/bin/bash
#file lsd
#provide disk usage of directories recursively in ls -lh listing

if [ $# -ne 1 ] #if no argument, assign pwd to var a
then
a='./'
else a=$1  #else assign argument to var a
fi

len=${#a} #extract length of a

#count char of a until the last occurrence of "/"
e=`expr "$a" : '.*/'`

#if count not equal to length of a
#(that is: argument without a final slash)
if [ "$e" -ne "$len" ]
then
a=${a}/  #add slash to argument
fi

#change the 5th field of "ls -lh" with "du -hs" 
ls -lh "$a" | awk -v dir_arg="${a}" '
     BEGIN {
     gsub(/ /,"\\ ",dir_arg)
     gsub(/'"'"'/,"'"\\\'"'",dir_arg)
     }
     substr($1,1,1)=="d"{
         {
         if ($3==$8 || $4==$8) 
         dir_listed = $8    
         else
         dir_listed = substr($0,index($0,$8))
         }
     gsub(/ /,"\\ ",dir_listed)
     gsub(/'"'"'/,"'"\\\'"'",dir_listed)
     ("du -hs "dir_arg dir_listed) | getline size
     split(size,size_)
     sub($5,size_[1],$5)
     }
     substr($1,1,1)=="t"{
     printf ("Files total size %s\n",$2)
     }
     substr($1,1,1)!="t" && substr($1,1,1)!="d"{
     printf ("%s %3s %s %s %5s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,substr($0,index($0,$8)))
     }
     substr($1,1,1)=="d"{
     printf ("%s %3s %s %s %5s %s %s %s\n",$1,$2,$3,$4,$5,$6,$7,dir_listed)
     }
     '
  
#print the whole dimension of argument
echo -n "Grand total size  "; du -sh "$a"

#updated 2011/01/14

It seems to me that it works.

Thanks to colucix for the suggested awk scripts.

Bye
gross6