perl script to choose top10 directiries!

ghostdog74 · 04-10-2009, 08:02 PM

Quote:

Originally Posted by Sergei Steshenko

So, you are a system administrator on a Linux box, you write a "junk" script - to something Q&D, or even a production script, but still for the Linux boxen, and you care about code portability ?

why not? you will never know about the future.

Quote:

but one has to know the real value/goal.

one will know the real value, when one has to port to a different environment and have to rewrite his code with much effort.

Sergei Steshenko · 04-11-2009, 09:42 AM

Quote:

Originally Posted by ghostdog74

why not? you will never know about the future.

one will know the real value, when one has to port to a different environment and have to rewrite his code with much effort.

The point is to choose effort properly.

When on command line I want to set PATH environment variable, I am using ':' as separator on my Linux box, not thinking about ';' Windows future.

A lot of scripts in system administrator life are written to be used just once, i.e. they are essentially a complicated command line.

Honestly, do you use '/' in your scripts, like

"$foo/$bar"

in shell/Perl terms, or generically write something like

Code:

compose_path
  (
  "/", # separator
  $foo,
  $bar
  )

?

AFAIR, there are OSes which have different from '/' path elements separator.

ghostdog74 · 04-11-2009, 11:25 AM

Quote:

Originally Posted by Sergei Steshenko

The point is to choose effort properly.

not my point though.

Quote:

When on command line I want to set PATH environment variable, I am using ':' as separator on my Linux box, not thinking about ';' Windows future.

that's why programming language (in my case:Python) provides things like
os.sep. eg on linux machine

Code:

>>> import os
>>> os.sep
'/'

on windows machine

Code:

C:\test> python
>>> import os
>>> os.sep
'\\'
>>>

the language takes care of that for me.

Quote:

A lot of scripts in system administrator life are written to be used just once, i.e. they are essentially a complicated command line.

not really. there are scripts written for system admin, but there are also scripts that are written to do back end jobs for applications. things may change due to business environment changes, therefore you will never know when one day you have to port code to different platform.

Quote:

Honestly, do you use '/' in your scripts, like

"$foo/$bar"

in shell/Perl terms, or generically write something like

Code:

compose_path
  (
  "/", # separator
  $foo,
  $bar
  )

?

it makes things easier in Perl if it has stuff like os.sep, right? but whatever it is, you can still make a custom version of "os.sep" in Perl.

Sergei Steshenko · 04-11-2009, 12:05 PM

Quote:

Originally Posted by ghostdog74

not my point though.

that's why programming language (in my case:Python) provides things like
os.sep. eg on linux machine

Code:

>>> import os
>>> os.sep
'/'

on windows machine

Code:

C:\test> python
>>> import os
>>> os.sep
'\\'
>>>

the language takes care of that for me.

not really. there are scripts written for system admin, but there are also scripts that are written to do back end jobs for applications. things may change due to business environment changes, therefore you will never know when one day you have to port code to different platform.

it makes things easier in Perl if it has stuff like os.sep, right? but whatever it is, you can still make a custom version of "os.sep" in Perl.

There is a bunch of modules in Perl with moral equivalent of "os.sep", but I honestly don't use them.

Because I write for UNIX-like OSes, and even on Windows '/' is still understood.

I see no chance, and so far my vision has been correct, that a business or myself switches to an existing OS with non-'/' separator.

ghostdog74 · 04-11-2009, 12:12 PM

Quote:

Originally Posted by Sergei Steshenko

Because I write for UNIX-like OSes, and even on Windows '/' is still understood.

yes, i think modern languages should be able to take care of that. "/" is also understood in this case.

Code:

>>> s="c:/test"
>>> os.chdir(s)
>>> os.getcwd()
'c:\\test'

Quote:

I see no chance, and so far my vision has been correct, that a business or myself switches to an existing OS with non-'/' separator.

so be it.

bigearsbilly · 04-11-2009, 04:41 PM

du -s * | sort -n

or a simple graphical view:
http://xdiskusage.sourceforge.net/

Sergei Steshenko · 04-11-2009, 05:35 PM

Quote:

Originally Posted by bigearsbilly

du -s * | sort -n

or a simple graphical view:
http://xdiskusage.sourceforge.net/

The OP has already been advised to use 'du' and 'sort', however, he has some kin of access problem, which might be due to automounter - I don't remember seeing his reply regarding automounter.

jimmyjiang · 05-17-2009, 10:50 AM

Quote:

Originally Posted by Telemachos

You will need to search recursively down, starting from two different places, and collect and add the sizes of the files you find. File::Find is good for that. Once you have the data all collected, you want to view the top ten only. That's easy enough to do: sort the data by total size, and then view the first ten from that sorted group.

I did something like this for work, and you're welcome to take a look at my two scripts and use them if they're helpful.

The first one builds the report and stores the output in a flatfile database. You can run this one every night, or multiple times a day, via a cron job.

The second one views the information. First it reads in the data from the stored database. The way I set it up, you can ask to view all the folders, sorted by size, or a certain number, but as the default (if you don't ask for anything special) you get the top ten by size.

My scripts are built to search through a bunch of folders starting from one specific place, and there will obviously be many details you won't want or will need to switch. Still, this might help you to get started. Good luck.

thanks for help!
when I run second script:
# perl space-user.pl
Can't locate Number/Bytes/Human.pm in @INC (@INC contains: /usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl/5.8.2 /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl/5.8.2 /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl .) at space-user.pl line 8.

what I should do?
thanks again!
jimmy

Sergei Steshenko · 05-17-2009, 11:12 AM

Quote:

Originally Posted by jimmyjiang

thanks for help!
when I run second script:
# perl space-user.pl
Can't locate Number/Bytes/Human.pm in @INC (@INC contains: /usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl/5.8.2 /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl/5.8.2 /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl .) at space-user.pl line 8.

what I should do?
thanks again!
jimmy

You need to install Number::Bytes::Human Perl module - either through your package manager or through 'cpan' command.

...

FWIW, your Perl is pretty outdated, and if you wish you can build the newest perl-5.10.0, and to build it portable/relocatable, i.e. you would be able to place the whole Perl directory tree wherever you like.

In such a manner you do not need to be root to install it, and it won't conflict with system-wide Perl.

jimmyjiang · 05-17-2009, 12:09 PM

I tried first script in my directory, it took long time to execute, never finished, I have to kill it manually, but when I execute it in a simple test directory, it works, generated "uploads.dat" file.
but when I execute second script( I named it space-user.pl) in my simple test directory, it gave me an error:
# perl space-user.pl
Can't locate Number/Bytes/Human.pm in @INC (@INC contains: /usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5 /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl/5.8.2 /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl/5.8.2 /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl .) at space-user.pl line 8.
BEGIN failed--compilation aborted at space-user.pl line 8.
what I should to?
thanks!
jimmy

jimmyjiang · 05-17-2009, 12:21 PM

my box is "Red Hat Enterprise Linux ES release 4 (Nahant Update 7)"
can you post the corresponding commands to install Number::Bytes::Human Perl module?
thanks a lot!

Sergei Steshenko · 05-17-2009, 01:06 PM

Quote:

Originally Posted by jimmyjiang

my box is "Red Hat Enterprise Linux ES release 4 (Nahant Update 7)"
can you post the corresponding commands to install Number::Bytes::Human Perl module?
thanks a lot!

I have never used this module. If it exists, just enter

Perl Number::Bytes::Human

into your favorite web search engine, and you'll get matches.

And/or visit http://search.cpan.org/ and enter

Number::Bytes::Human

there.

Telemachos · 05-17-2009, 07:47 PM

Quote:

Originally Posted by jimmyjiang

I tried first script in my directory, it took long time to execute, never finished, I have to kill it manually, but when I execute it in a simple test directory, it works, generated "uploads.dat" file.

I posted the scripts for you to look at as examples. They cannot possibly work on your machine as is, if for no other reason than that they don't have a proper place to start from. If you look at the first script, it currently sets the base directory (the place you start your search from) in this way:

Code:

my $base_dir = '/path/to/start';

I removed the actual path from my script and put that there as a way of telling you "Edit this bit for yourself." You need to change that file at the very least to put a proper path there. (I'm actually confused that the script runs at all. If I try it here, as is, I get this:

Code:

hektor ~/Desktop $ perl 874.pl
Can't open /path/to/start: No such file or directory

That's a deliberate choice in the script. If it can't find the base directory, it dies with an error message. Did you edit that file already?)

Quote:

Originally Posted by jimmyjiang

but when I execute second script( I named it space-user.pl) in my simple test directory, it gave me an error:
# perl space-user.pl
Can't locate Number/Bytes/Human.pm in @INC
<snip>
BEGIN failed--compilation aborted at space-user.pl line 8.

As Segei said, you will need to install the Number::Bytes::Human module from the CPAN. (I use it a lot for this sort of task. It provides a simple way to translate byte counts into something a bit more human.) If you've never used CPAN, I recommend that you start at this site.

But the most important thing is that those scripts are examples of working Perl code. You shouldn't necessarily expect to be able to use them on a completely different machine without some tweaking.

Edit: In terms of performance, I just tested it out on my system at $WORK. I have 65 blogs with between 0 and over 200 items in their uploads folders (and sub-folders). Here's how long it took to run the script:

Code:

admin ~/build/scripts $ time ./uploads_report 

real    0m3.029s
user    0m0.680s
sys     0m1.162s

Now performance in the real word always has a lot of subtle variations, but unless you're dealing with an immense number of items, the script shouldn't hang your system up for an extended period of time.