LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-10-2012, 03:56 AM   #1
patrick295767
Member
 
Registered: Feb 2006
Distribution: FreeBSD, Linux, Slackware, LFS, Gparted
Posts: 664

Rep: Reputation: 138Reputation: 138
robust total size of a list of files


Hi,

There are different methods but what is the most robust method to make the total size of the files and/or directories from a list of files (`ls -1 `)?

Code:
ls -1 | while read -r each ; do 

tmpsize=$(du -b "${each}")
totalsize=??

done

echo "The total size of the list is: $totalsize"


thanks
 
Old 05-10-2012, 04:04 AM   #2
em31amit
Member
 
Registered: Apr 2012
Location: /root
Distribution: Ubuntu, Redhat, Fedora, CentOS
Posts: 190

Rep: Reputation: 55
du also shows total count of file sizes


check "-c" parameters in du man pages.
 
Old 05-10-2012, 04:11 AM   #3
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
Quote:
Originally Posted by patrick295767 View Post
Hi,

There are different methods but what is the most robust method to make the total size of the files and/or directories from a list of files (`ls -1 `)?

Code:
ls -1 | while read -r each ; do 

tmpsize=$(du -b "${each}")
totalsize=??

done

echo "The total size of the list is: $totalsize"


thanks
You really don't want to use ls for this. What's the exact
requirement of this exercise?
 
Old 05-10-2012, 05:27 AM   #4
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
For a list of files, I'd use
Code:
stat -c '%s' FILE(s).. | awk '{ s=s+$1 } END { printf("%.0f\n", s) }'
but for complete directory trees, I'd use
Code:
find DIR(S).. -type f -printf '%s\n' | awk '{ s=s+$1 } END { printf("%.0f\n", s)
If you want the output in millions of bytes, with one decimal digit, use printf("%.1f MB\n", s/1000000.0)

Remember that du lists disk usage, not file sizes.
 
Old 05-10-2012, 05:36 AM   #5
em31amit
Member
 
Registered: Apr 2012
Location: /root
Distribution: Ubuntu, Redhat, Fedora, CentOS
Posts: 190

Rep: Reputation: 55
@Nominal

du lists disk usage, but by whom?? ofcourse a file.. everything is file in linux. than what is the difference between size listed by du for a file and file size showing using stat command ?
 
Old 05-10-2012, 06:53 AM   #6
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Quote:
Originally Posted by em31amit View Post
du lists disk usage
Exactly. Observe:
Code:
dd if=/dev/zero of=example-file bs=1 count=1 seek=1048574
This produces a sparse file. You can read it normally, but the initial zeros you see are not stored on disk, and do not consume disk space.

So, even though ls -l example-file outputs
Code:
-rw-rw-r-- 1 nominal animal 1048575 May 10 14:43 example-file
and find example-file -printf '%s\n' and stat -c '%s' example-file both output
Code:
1048575
the actual disk usage is much less than the file size. My ext4 partition I used for this example utilizes 4k blocks, so du -hs example-file outputs
Code:
4.0K	example-file
Do you now see the difference between file sizes and disk usage?

Directories also consume disk space. The amount depends on the number of entries, the length of the file and directory names in it, and the number of POSIX and extended attributes used. If the du command you run includes the directories, the result will reflect the actual disk space used, and will be significantly different to the total size of the relevant files. You cannot even say which one (disk usage or total file size) is greater, unless you check!
 
1 members found this post helpful.
Old 05-10-2012, 07:23 AM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 10,008

Rep: Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193Reputation: 3193
Quote:
echo "The total size of the list is: $totalsize"
Someone should also point out that this line will never work as expected seeing the while loop is in a sub-shell and so any changes to the variables inside the loop
will not be reported outside the loop.
 
Old 05-10-2012, 07:24 AM   #8
millgates
Member
 
Registered: Feb 2009
Location: 192.168.x.x
Distribution: Slackware
Posts: 852

Rep: Reputation: 389Reputation: 389Reputation: 389Reputation: 389
Quote:
Originally Posted by Nominal Animal View Post
Exactly. Observe:
Code:
dd if=/dev/zero of=example-file bs=1 count=1 seek=1048574
This produces a sparse file. You can read it normally, but the initial zeros you see are not stored on disk, and do not consume disk space.

So, even though ls -l example-file outputs
Code:
-rw-rw-r-- 1 nominal animal 1048575 May 10 14:43 example-file
and find example-file -printf '%s\n' and stat -c '%s' example-file both output
Code:
1048575
the actual disk usage is much less than the file size. My ext4 partition I used for this example utilizes 4k blocks, so du -hs example-file outputs
Code:
4.0K	example-file
Do you now see the difference between file sizes and disk usage?

Directories also consume disk space. The amount depends on the number of entries, the length of the file and directory names in it, and the number of POSIX and extended attributes used. If the du command you run includes the directories, the result will reflect the actual disk space used, and will be significantly different to the total size of the relevant files. You cannot even say which one (disk usage or total file size) is greater, unless you check!
That is correct. However, look at the -b and --apparent-size switches for du which will show the correct result.
 
Old 05-10-2012, 08:50 AM   #9
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Mint, Armbian, NetBSD, Puppy, Raspbian
Posts: 3,515

Rep: Reputation: 239Reputation: 239Reputation: 239
What about?

cat * | wc -c

well typically to compare directories I do

Code:
cat A/* | cksum
cat B/* | cksum
but I am
a) lazy
b) a guru
c) lateral thinking

not sure which
 
1 members found this post helpful.
Old 05-10-2012, 10:37 AM   #10
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Quote:
Originally Posted by millgates View Post
That is correct. However, look at the -b and --apparent-size switches for du which will show the correct result.
Only if the parameter list does not refer to any directories. When using
Code:
 du -b --apparent-size ... DIRECTORY ...
the result includes the size of the directories, not just the files. It is therefore not the total size of the files, as specified in the title of this thread; it yields the total size of the specified files and directories.
 
Old 05-10-2012, 11:41 AM   #11
millgates
Member
 
Registered: Feb 2009
Location: 192.168.x.x
Distribution: Slackware
Posts: 852

Rep: Reputation: 389Reputation: 389Reputation: 389Reputation: 389
Quote:
Originally Posted by Nominal Animal View Post
Only if the parameter list does not refer to any directories. When using
Code:
 du -b --apparent-size ... DIRECTORY ...
the result includes the size of the directories, not just the files. It is therefore not the total size of the files, as specified in the title of this thread; it yields the total size of the specified files and directories.
Isn't a directory also a file (everything's a file in unix)? By the way, the original question was (emphasis added):

Quote:
There are different methods but what is the most robust method to make the total size of the files and/or directories from a list of files (`ls -1 `)?
if you don't want directories, you can use find . -type f or whatever.
 
Old 05-10-2012, 12:32 PM   #12
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Quote:
Originally Posted by millgates View Post
Isn't a directory also a file
There are enough differences that make the distinction meaningful.

Directory size is almost always an integral number of disk allocation blocks, making it useless for detecting e.g. additions or removals. (Note: Symlink size is almost always the length of the symlink target path.)

Directories cannot be hardlinked, only files can. (You can, however, bind-mount directories and files over other directories and files, so only the "top one" is accessible.)

Execute right to a directory allows you to stat() the directory and any file you know the name of in that directory. (Without read rights, you cannot scan the directory, so I guess read and write rights are analogous between files and directories.)

For mount points, /path/to/directory and /path/to/directory/. have different device and inode numbers.
 
Old 05-20-2012, 01:57 PM   #13
patrick295767
Member
 
Registered: Feb 2006
Distribution: FreeBSD, Linux, Slackware, LFS, Gparted
Posts: 664

Original Poster
Rep: Reputation: 138Reputation: 138
thank you so so so much !!!!!

I like the --apparent-size way and too the find frmo unspawn
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] [Lubuntu] Files' Size On Disk is ~ 10x Total Size Of Files! Zssfssz Linux - Software 2 02-22-2012 09:16 PM
Total particular files size in a directory kirukan Linux - Newbie 3 01-03-2010 02:48 PM
finding size of total files in a directory blackzone Linux - Newbie 3 01-07-2005 03:01 AM
How to list total file size of a directory phil1076 Linux - General 3 12-18-2003 03:47 PM
Command to list total number of files. WillieB_72 Linux - General 3 01-29-2003 09:25 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 02:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration