LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-01-2011, 10:29 PM   #1
sysmicuser
Member
 
Registered: Mar 2010
Posts: 332

Rep: Reputation: 0
Unhappy Total space occupied by all files above 1G


Hi,
I am doing something like this.

[user01@tmelbld01 dropzone]$find . -size +1G -exec ls -larth {} \;|awk '{print $5}'
find: ./DM/lost+found: Permission denied
2.7G
4.7G
2.0G
find: ./1.9.0.6/Hotfix/QC16841/patch_16841: Permission denied
find: ./1.9.0.6/Hotfix/QC16847/CA: Permission denied
4.1G
1.5G
2.9G
1.8G
1.8G
1.8G
1.9G
4.2G
1.9G
4.2G
1.9G
4.3G
2.0G
2.0G
find: ./lost+found: Permission denied
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
2.0G
1.5G
1.1G
1.1G
1.1G
1.1G
1.2G
1.1G
1.8G
1.8G
1.8G
1.9G
1.8G
1.8G
1.8G
1.8G
1.8G
1.8G
1.8G
1.8G
1.8G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.1G
1.5G
1.5G
1.1G
2.4G
1.1G
[user01@tmelbld01 dropzone]$

What I want is "no entries where it says permission denied" + add sum of first colums mena s 1.1G, 2.4G etc total GB should be dispalyed at last colum with Total:

Is this possible?

Please educate.

Many thanks
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 10-02-2011, 01:44 AM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Code:
 find . -size +1G -exec ls -lart {} \; 2>/dev/null|awk '{size+=$5}END{print size/1024/1024/1024"G"}
[edit]
I should probably explain why I took that 'h' off from ls'
output .... rounding errors. Using size in bytes, adding
up and dividing as I did yields a result out by > 4 GB on
my machine, thought that's worthwhile considering[/edit]


Cheers,
Tink

Last edited by Tinkster; 10-02-2011 at 02:06 AM. Reason: [edit][/edit]
 
2 members found this post helpful.
Old 10-02-2011, 09:58 AM   #3
sysmicuser
Member
 
Registered: Mar 2010
Posts: 332

Original Poster
Rep: Reputation: 0
Smile

Thank you very much for your reply, that indeed worked and gave me result at 106GB, that amount of space we are "wasting".

Now I am trying to understand the logic behind your one line.

Please help me understand.

First bit,
find . -size +1G -exec ls -lart {} \;
this is list all files in long format and sorting as per rt.
then,
2>/dev/null
here you are re-directing output "standard output" to null device.

the last bit I am unable to comprehend,
awk '{size+=$5}END{print size/1024/1024/1024"G"}'
here you are printing 5th column and start evaluation from 0 I think
you wrote END but where is BEGIN?

Also on a side note when I do find for some files I do get "Permission denied" your one liner is ignoring that? I believe.

Please educate.

Many Thanks in advance.
 
Old 10-02-2011, 02:26 PM   #4
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by sysmicuser View Post
Thank you very much for your reply, that indeed worked and gave me result at 106GB, that amount of space we are "wasting".

Now I am trying to understand the logic behind your one line.

Please help me understand.

First bit,
find . -size +1G -exec ls -lart {} \;
this is list all files in long format and sorting as per rt.
Yup.. That said, I should have trimmed the fat; since find will
run the ls against individual files sorting is pretty pointless
in this case. Make that "ls -la"

Quote:
Originally Posted by sysmicuser View Post
then,
2>/dev/null
here you are re-directing output "standard output" to null device.
Nope, standard error. You didn't want to see the "permission denied",
and that's what we suppress.

Quote:
Originally Posted by sysmicuser View Post
the last bit I am unable to comprehend,
awk '{size+=$5}END{print size/1024/1024/1024"G"}'
here you are printing 5th column and start evaluation from 0 I think
you wrote END but where is BEGIN?
BEGIN is not required in this case. awk operates "per line" of
text/output/... so what we do is: we create a variable "size".
For each line of find's ls output we add the 5th column to it.
After all lines have been processed (END block) we print the
added up values.




Cheers,
Tink
 
Old 10-02-2011, 02:59 PM   #5
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,606

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Quote:
Originally Posted by Tinkster View Post
Code:
 find . -size +1G -exec ls -lart {} \; 2>/dev/null|awk '{size+=$5}END{print size/1024/1024/1024"G"}
[edit]
I should probably explain why I took that 'h' off from ls'
output .... rounding errors. Using size in bytes, adding
up and dividing as I did yields a result out by > 4 GB on
my machine, thought that's worthwhile considering[/edit]


Cheers,
Tink
Two suggestions, if I may add:
1) using 'stat -c "%s" {} ' instead of 'ls -lart {}'. The output of 'ls' might differ on different systems. Using 'stat' makes a small modification to 'awk' necessary:
awk '{size+=$0}END{print size/1024/1024/1024"G"}

2) adding '-type f' to find. This does not have any immediate impact if find operates on this scale ( > 1G ). But if one wishes to change that, e.g. to > 1k then also directories would be found. And especially with 'ls' this would yield highly erroneous results.

Code:
find . -type f -size +1G -exec stat -c "%s" {} \; 2>/dev/null|awk '{size+=$0}END{print size/1024/1024/1024"G"}
 
Old 10-02-2011, 03:52 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by crts View Post
Two suggestions, if I may add:
1) using 'stat -c "%s" {} ' instead of 'ls -lart {}'. The output of 'ls' might differ on different systems. Using 'stat' makes a small modification to 'awk' necessary:
awk '{size+=$0}END{print size/1024/1024/1024"G"}
Can't say I've come across an ls on any posix compliant OS that would
yield different byte sizes for files from stat, your mileage may vary.


Point 2 is quite valid.


Cheers,
Tink
 
Old 10-02-2011, 05:59 PM   #7
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,606

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Quote:
Originally Posted by Tinkster View Post
Can't say I've come across an ls on any posix compliant OS that would
yield different byte sizes for files from stat, your mileage may vary.
My concern was not so much that the size itself would be reported wrongly but that field 5 *might* not always contain file size.
I once read here
http://www.grymoire.com/Unix/Awk.html#uh-2

that on Linux systems the output differs from unix systems. Not sure if size is affected but it seems that there were (still are?) different implementations of 'ls' at some time.
 
Old 10-15-2011, 08:25 PM   #8
sysmicuser
Member
 
Registered: Mar 2010
Posts: 332

Original Poster
Rep: Reputation: 0
Thanks to both of you for your help.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to obtain the total size occupied by files of certain type? jstitch Linux - Software 2 08-03-2010 04:28 PM
[SOLVED] How can I find the total disk space used by all files with the same extension (*.dat) respex Linux - Newbie 5 12-04-2009 07:38 AM
Total available/used space of all drives MartenH Linux - Newbie 16 03-29-2009 05:04 PM
Nmap help, total newblet from space. zarathaz Linux - Newbie 3 07-20-2005 12:49 AM
format only gives me less than total space ctrl-a Linux - Newbie 4 10-12-2004 05:15 PM


All times are GMT -5. The time now is 09:27 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration