LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-06-2011, 04:05 PM   #1
smritisingh03
Member
 
Registered: Nov 2010
Posts: 43

Rep: Reputation: 0
how to speed up this script


Hi

my script is running very slow.It basically compares 2 output files.Please suggest.URGENT!!!

#!/bin/sh


compareLOGandDBfunction()
{

export file1="DBcountOP"
export file2="logcountOP"
rm -f comp_res2.txt
rm -f temp2.txt
rm -f result2.txt
match=0
export comp_res="comp_res2.txt"
export temp="temp2.txt"
export result="compareLOGandDB_OP.txt"

while read FILE1_LINE ; do
#reading the output file from database and the first string is stored in the variable LINE


file1_tablename="$(echo $FILE1_LINE | cut -d ' ' -f1)"
#the 1st field is stored i.e,tablename

file1_count="$(echo $FILE1_LINE | cut -d ' ' -f2)"
#the 2nd filed is stored i.e,the count of rows for the table


while read FILE2_LINE ; do
#reading the OP file from log and stores the 1st string



file2_tablename="$(echo $FILE2_LINE | cut -d ' ' -f1)"
#the 1st field is stored i.e,tablename

file2_count="$(echo $FILE2_LINE | cut -d ' ' -f2)"
#the 2nd fild stored-the count of rows for the table


if [ "$file1_tablename" = "$file2_tablename" ] && [ "$file1_count" = "$file2_count" ]
#start of 1st if block

then
echo table has matched $file1_tablename
echo count has matched $file2_count
echo $file1_tablename " " $file1_count " " $file2_count "\t" Matched | tee -a $temp $result


fi #end of 1st if block

if [ "$file1_tablename" = "$file2_tablename" ] && [ "$file1_count" != "$file2_count"]
#start of 2nd if
#checking for match b/w tables but a mismatch b/w the rowcount from logfile and the DB output file


then
echo table has matched $file1_tablename
echo count has not matched $file2_count
echo $file1_tablename " " $file1_count " " $file2_count "\t" Count MissMatch | tee -a $temp $result
fi
#end of 2nd if block

done < $file2


done < $file1

echo *******************************************************

#the following 2 blocks check for tables that exists only in either of the logfile or the database

while read FILE2_LINE ; do
#reading the logfile line after line


file2_tablename="$(echo $FILE2_LINE | cut -d ' ' -f1)"


file2_count="$(echo $FILE2_LINE | cut -d ' ' -f2)"
echo table name is $file2_tablename
echo count is $file2_count
match=`grep $FILE2_LINE $temp | wc -l`
if [ $match -eq 0 ]
then
echo $FILE2_LINE not present in $file1 | tee -a $comp_res $result


fi
#checks for a table that may exist in log but not in the database

done < $file2

echo ******************************************************


while read FILE1_LINE ; do
#reading the output file from the database

file1_tablename="$(echo $FILE1_LINE | cut -d ' ' -f1)"
file1_count="$(echo $FILE1_LINE | cut -d ' ' -f2)"
echo table name is $file1_tablename
echo count is $file1_count
match=`grep $FILE1_LINE $result | wc -l`
if [ $match -eq 0 ]
then
echo $FILE1_LINE not present in $file2 | tee -a $comp_res $result
fi
#checks for a table that might exist in database but not in the log
done < $file1
 
Old 01-06-2011, 04:10 PM   #2
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
Quote:
Originally Posted by smritisingh03 View Post
Please suggest.URGENT!!!
Hi,

this is only urgent to you. We are all volunteers here and we will help when we have the time and when we want to. So please do not use terms like urgent in your posts.
 
Old 01-06-2011, 09:06 PM   #3
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,423

Rep: Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823Reputation: 2823
Well the first part with the 2 while loops looks like a complicated way to do a simple diff?? Any reason this tool cannot work for you?
 
Old 01-06-2011, 09:09 PM   #4
smritisingh03
Member
 
Registered: Nov 2010
Posts: 43

Original Poster
Rep: Reputation: 0
Wink

Hi there...btw i know someone would answer my query only if he has time...in fact all of us are aware of this!!! URGENT doesnt mean that i need attention to this code even if anyone doesnt have time....it is just to increase its priority...hope u understand this and would be a bit courteous...
 
Old 01-07-2011, 02:41 AM   #5
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,608

Rep: Reputation: 448Reputation: 448Reputation: 448Reputation: 448Reputation: 448
some simple rules ...

Quote:
Originally Posted by smritisingh03 View Post
URGENT doesnt mean that i need attention ...
Marking your post as URGENT is still considered rude.
Quote:
it is just to increase its priority ...
This is where you are mistaken. You need to understand that you have no means to prioritize your thread. I know that there are other sites that actually do have dedicated sub-forums for urgent requests. LQ, however, does not implement such a feature (yet). And 'urgent' requests tend to get less responses. So if someone points out to you that 'urgent' requests are frowned upon then it is in your best interest.

Two other points of notice:
1)
Quote:
Hi there...btw i know someone would answer my query only if he has time...in fact all of us are aware of this!!! URGENT doesnt mean that i need attention to this code even if anyone doesnt have time....it is just to increase its priority...hope u understand this and would be a bit courteous...
Proper spelling, capitalization and punctuation is also highly appreciated by most members. Text-speak language is hard to read and it gets annoying after a while.

2)
When you are posting code samples then you should enclose them in code-tags. This way your code stays properly formatted and is easier to read. See my signature on how to use code-tags.

I hope you understand that by abiding to this simple rules your threads will get better resonance here on LQ.
 
Old 01-07-2011, 11:17 AM   #6
w1k0
Senior Member
 
Registered: May 2008
Location: Poland
Distribution: Slackware, Mint
Posts: 1,252

Rep: Reputation: 205Reputation: 205Reputation: 205
smritisingh03,

From a reader point of view your script is long and uses a lot of unknown files. It’s difficult to guess what it should do. I suppose you know that. But I don’t know.

Moreover your script is invalid. The function begins:

Code:
compareLOGandDBfunction()
{
but never ends.

I suggest you to put at the beginning of the script the following code:

Code:
started=$(date +"%s")

function works {
elapsed=$(expr $(date +"%s") - $started)
started=$(expr $started + $elapsed)
echo -e "\033[1m$elapsed sec.\033[0m"
echo
}
Then put works command in selected places to track the time passed from the start of the script, for example:

Code:
.
.
.
done < $file2
works
.
.
.
The other method is to put at the beginning of the script the following code:

Code:
declare -i number=0

function number {
number=$number+1
echo
echo -e "\033[1m$number.\033[0m"
}
Then precede the commands you suspect work too slow with the sequence number; time, for example:

Code:
.
.
.
number; time while read FILE1_LINE ; do
.
.
.

Last edited by w1k0; 01-07-2011 at 11:29 AM. Reason: improvement of the code
 
Old 01-07-2011, 12:05 PM   #7
smritisingh03
Member
 
Registered: Nov 2010
Posts: 43

Original Poster
Rep: Reputation: 0
Thankyou so much...i could find out where the problem is.Actually this script alone is wkng fine.I used ur code for the script which is querying the DB and there the culprit was :-).

Now another question....i have a script which is doing a "select * from tab;" and writing the O/P into a file x2.

then in the second part I am using this file x2 as my input ,reading with a while loop and querying each table for its rowcount "select count(*) from tablename" .

There are possibly n*1000 tables in DB and that is the reason why it is taking so long!!

Does anyone here have a clue as to how can I improve this.

Actually I want to writ a function for the sql part and then use a variable maybe "maxparellel=5" and then call the sql function in a while loop and execute it using "&".The moment it processes 5 rows parellely i give the wait command.

please let me know if this is a good workaround and please help me with this.I need to know how to go ahead!!!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Bandwidth management: speed 128kbps, after 500MB, speed down to 64kbps zdienos Linux - Networking 2 01-27-2009 08:07 AM
Application / Script to test READ speed on a filesystem BradRalston Linux - Software 1 05-10-2008 03:01 AM
NIC transfer speed fast... receive speed slow landev Linux - Networking 5 11-07-2006 04:09 PM
cdrecord - trying to use high speed medium on low speed writer captain-cat Linux - Hardware 2 07-12-2004 07:27 PM
Is there a tool to monitor Internet connection speed and also network speed? xleft4dexy Linux - Networking 4 10-14-2003 11:29 PM


All times are GMT -5. The time now is 05:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration