LinuxAnswers - the LQ Linux tutorial section.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 11-02-2011, 11:58 AM   #1
LQ Newbie
Registered: Nov 2011
Posts: 1

Rep: Reputation: Disabled
Question Small script optimization

Hello everyone,

I have a simple script that searches for folders below the base parameter and prints the quantity of files inside them.

The problem is that I have around 900.000 files in some folders and this process is very slow.

DIRS=$(find "$base" -type d)
for d in $DIRS
        echo "$d         $(find $d -type f | wc -l)"
I am not a linux experienced user, so, if any of you have any ideas to make it perform faster it's very appreciated.

Old 11-02-2011, 12:54 PM   #2
Senior Member
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,170

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
Here's another approach that might be faster:
[ -d "${1}" ] && echo \"${1}\" is not a directory. Aborting. >&2 && exit 1
sudo updatedb -U "${1}" -o /tmp/"${1}.db"
echo "$(locate -Sd /tmp/"${1}.db" | grep files) exist in or below ${1}."
# rm -f /tmp/"${1}.db"
Note 1: This is untested code.
Note 2: The deletion of the temporary data base file is commented out because you might find the locate command useful for other reasons, and you might, therefore, want to keep it around. The creation of the db file in /tmp is, of course, arbitrary. It could be placed anywhere you wanted it, although placing in the the tree to wanted to count might be counter-productive.

Here's a version that worked for me:
if [ $# -lt 1 ] || [ "${1,,*}" == "-h" ] || [ "${1,,*}" == "--help" ]
  cat <<EOF >&2
$0: Count the number of files in or below a specified directory.

Argument: Root directory
[ ! -d "${1}" ] && echo \"${1}\" is not a directory. Aborting. >&2 && exit 1
tmpfile=$(mktemp /tmp/locdb-XXXXX)
sudo updatedb -U "${1}" -o ${tmpfile}
echo "$(locate -Sd ${tmpfile} | grep files) exist in or below ${1}."
#rm -f ${tmpfile}

Last edited by PTrenholme; 11-02-2011 at 01:50 PM. Reason: Typo
1 members found this post helpful.
Old 11-02-2011, 07:16 PM   #3
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,626

Rep: Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149
If that isn't fast enough, consider Perl. It calls the underlying C libs directly and runs almost as fast as C (its compiled on the fly before being run). Should be much quicker than calling shell level programs.


bash, find, loop, script

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Building Firefox 3.6 - A "Profile Guided Optimization" (PGO) Automated Build Script GrapefruiTgirl Linux - Software 6 10-28-2010 09:50 AM
BASH script optimization for testing large number of files instag Programming 24 09-26-2010 11:40 PM
small script spx2 Linux - Newbie 10 12-13-2005 12:35 PM
help with a very small script hamish Linux - General 3 06-09-2004 02:10 AM
Please Help with a small script ? juglugs Programming 2 11-14-2001 02:39 AM

All times are GMT -5. The time now is 07:58 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration