LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-18-2012, 07:57 AM   #1
steven.c.banks
Member
 
Registered: Dec 2007
Location: Virginia
Distribution: RHEL
Posts: 44
Blog Entries: 1

Rep: Reputation: 15
Bash shell array processing slower than expected


I have a bash script on Solaris to identify the last login date for an application. It extracts an ID from the first array, and searches for it in the second array, using two nested while do done loops.

The script works as designed, but is much slower than I anticipated. I have commented out portions to try to identify what makes it slow but have not identified anything. I would have thought that since everything is in memory it would run more quickly than it does (about three hours where there are 917 entries in idarray and 558 entries in loginarray). The two arrays are unfortunately not in the same sequence.

Thanks in advance...

Code:
while [ $idarraycounter -lt $idarrayrows ]
do
  id=`echo ${idarray[$idarraycounter]}`
  loginarraycounter=0
  lastlogindate=""
  while [ $loginarraycounter -lt $loginarrayrows ]
  do
    loginid=`echo ${loginarray[$loginarraycounter]} | cut -f1 -d","`
    loginidlower=`echo $loginid | tr '[:upper:]' '[:lower:]'`
    loginidupper=`echo $loginid | tr '[:lower:]' '[:upper:]'`
    if [[ "$loginid" = "$id" || "$loginidlower" = "$id" || "$loginidupper" = "$id" ]]
    then
      lastlogindate=`echo ${loginarray[$loginarraycounter]} | cut -f2 -d","`
      loginarraycounter=$loginarrayrows
    else
      ((loginarraycounter++))
    fi
  done
  # Write ID,lastlogindate
  # Note - if ID is not found (i.e. no logins), $lastlogindate will still be blank
  echo "$id,$lastlogindate"
  ((idarraycounter++))
done
 
Old 04-18-2012, 08:38 AM   #2
crabboy
Moderator
 
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,823

Rep: Reputation: 120Reputation: 120
Your script looks pretty good, but you are spawning a lot of processes looping over these arrays. You can try to eliminate some of them to speed it up. Try a case insensitive compare instead of spawning the 'tr's for the upper and lower compare. There is an example here:
http://www.linuxquestions.org/questi...sitive-676101/

You may also want to try to split the loginarray into two separate arrays, one with field 1 and the second with field2, this would eliminate the cut in the inner loop.

The time still seems quite long for what you are doing. I generally move to a better language when there is a need for arrays in a bash script.

Last edited by crabboy; 04-18-2012 at 08:40 AM.
 
1 members found this post helpful.
Old 04-18-2012, 08:39 AM   #3
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,645

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
you invoke at least 8 additional processes inside the double loop. that is not really efficient.
you can try to do the same with only one awk or perl script, and you will see the difference
 
1 members found this post helpful.
Old 04-18-2012, 09:44 AM   #4
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
I say don't use arrays, bash is not good at that. Use grep or sed or awk on files. Do you have input files or just arrays ?
 
1 members found this post helpful.
Old 04-18-2012, 12:15 PM   #5
wpeckham
Member
 
Registered: Apr 2010
Location: USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix
Posts: 765

Rep: Reputation: 172Reputation: 172
I would use

My goto tool for something like this would be Perl. It is made for things like this, and would eliminate all external calls.

There are some BASH tricks that would reduce external calls and speed your processing, but I really think porting to perl would result in an order of magnitude improvement.
 
1 members found this post helpful.
Old 04-19-2012, 01:51 PM   #6
steven.c.banks
Member
 
Registered: Dec 2007
Location: Virginia
Distribution: RHEL
Posts: 44
Blog Entries: 1

Original Poster
Rep: Reputation: 15
Thanks for your replies. After studying awk for a while, I decided to replace the entire inner loop with:

lastlogindate=`cat /u01/export/reports/login-ids-and-dates-master.csv | grep -i "^$id," | cut -f2 -d","`

where /u01/export/reports/login-ids-and-dates-master.csv is the file that I used to load the inner array. I thought catting the file 900 times would be slow, but the script now runs in less than one minute.

Thanks again...
 
Old 04-20-2012, 01:10 AM   #7
pan64
Senior Member
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 4,645

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
even, you do not need cat:
lastlogindate=`grep -i "^$id," /u01/export/reports/login-ids-and-dates-master.csv | cut -f2 -d","`
will also work
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Slackware 64 1337, slight delay when processing commands using Bash shell hyperhead Slackware 7 04-26-2011 03:05 AM
[SOLVED] reading unix file permissions into a bash array for processing dazdaz Programming 11 12-06-2010 03:21 PM
Network slower than expected, how do I detect the cause? MikeyCarter Linux - Networking 2 06-25-2008 02:29 PM
Bash Shell Scripting - using ls into array aliasofmike Programming 5 11-05-2007 03:00 PM
processing a ksh shell in bash environment ntoughe Programming 9 09-12-2005 05:09 AM


All times are GMT -5. The time now is 05:45 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration