LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
LinkBack Search this Thread
Old 11-02-2011, 11:58 AM   #1
tunilopez
LQ Newbie
 
Registered: Nov 2011
Posts: 1

Rep: Reputation: Disabled
Question Small script optimization


Hello everyone,

I have a simple script that searches for folders below the base parameter and prints the quantity of files inside them.

The problem is that I have around 900.000 files in some folders and this process is very slow.

Code:
#!/bin/bash
base=$1
DIRS=$(find "$base" -type d)
for d in $DIRS
do
        echo "$d         $(find $d -type f | wc -l)"
done
I am not a linux experienced user, so, if any of you have any ideas to make it perform faster it's very appreciated.

Thanks
 
Old 11-02-2011, 12:54 PM   #2
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,141

Rep: Reputation: 329Reputation: 329Reputation: 329Reputation: 329
Here's another approach that might be faster:
Code:
#!/bin/bash
[ -d "${1}" ] && echo \"${1}\" is not a directory. Aborting. >&2 && exit 1
sudo updatedb -U "${1}" -o /tmp/"${1}.db"
echo "$(locate -Sd /tmp/"${1}.db" | grep files) exist in or below ${1}."
# rm -f /tmp/"${1}.db"
Note 1: This is untested code.
Note 2: The deletion of the temporary data base file is commented out because you might find the locate command useful for other reasons, and you might, therefore, want to keep it around. The creation of the db file in /tmp is, of course, arbitrary. It could be placed anywhere you wanted it, although placing in the the tree to wanted to count might be counter-productive.

<edit>
Here's a version that worked for me:
Code:
#!/bin/bash
if [ $# -lt 1 ] || [ "${1,,*}" == "-h" ] || [ "${1,,*}" == "--help" ]
then
  cat <<EOF >&2
$0: Count the number of files in or below a specified directory.

Argument: Root directory
EOF
  exit
fi
[ ! -d "${1}" ] && echo \"${1}\" is not a directory. Aborting. >&2 && exit 1
tmpfile=$(mktemp /tmp/locdb-XXXXX)
sudo updatedb -U "${1}" -o ${tmpfile}
echo "$(locate -Sd ${tmpfile} | grep files) exist in or below ${1}."
#rm -f ${tmpfile}

Last edited by PTrenholme; 11-02-2011 at 01:50 PM. Reason: Typo
 
1 members found this post helpful.
Old 11-02-2011, 07:16 PM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5
Posts: 16,086

Rep: Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995
If that isn't fast enough, consider Perl. It calls the underlying C libs directly and runs almost as fast as C (its compiled on the fly before being run). Should be much quicker than calling shell level programs.
 
  


Reply

Tags
bash, find, loop, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Building Firefox 3.6 - A "Profile Guided Optimization" (PGO) Automated Build Script GrapefruiTgirl Linux - Software 6 10-28-2010 09:50 AM
BASH script optimization for testing large number of files instag Programming 24 09-26-2010 11:40 PM
small script spx2 Linux - Newbie 10 12-13-2005 12:35 PM
help with a very small script hamish Linux - General 3 06-09-2004 02:10 AM
Please Help with a small script ? juglugs Programming 2 11-14-2001 02:39 AM


All times are GMT -5. The time now is 09:51 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration