[SOLVED] make this script faster?

shakazzolo · 07-13-2010, 02:16 PM

This is a script that I use to cleanup the /lib/firmware and /lib/modules but its really slow... is there any smarter way for doing this?

I can only run bash scripts and compiled programs (C/C++) on that system... so python or perl is not an option

Code:

export mymod=`lsmod | awk -F' ' '{print $1}' | grep -v Module`
for file in `find . -type f`
 do
 testResult=""
 for mod in $mymod
   do
	tmpVar=`echo $file | grep -i $mod`
	if [ -n "$tmpVar" ]; then
		testResult=$mod
		break
	fi
   done

 if [ -n "$testResult" ]; then
	printf "$file is used by $testResult \n"
 else
	printf "Deleting $file\n"
	rm -Rf $file
 fi
done

Sergei Steshenko · 07-13-2010, 03:00 PM

Quote:

Originally Posted by shakazzolo

...
I can only run bash scripts and compiled programs (C/C++) on that system... so python or perl is not an option

...

Both Perl and Python are compiled programs.

unSpawn · 07-13-2010, 04:30 PM

Quote:

Originally Posted by Sergei Steshenko

Both Perl and Python are compiled programs.

Segei, please realize when you post you're supposed to help out people. Having only ten posts here this user may or may not have the amount of knowledge (yet) a pro possesses or might have phrased things differently than a pro would. So do look beyond that and please ditch the pedantic tone of voice.

Sergei Steshenko · 07-13-2010, 04:46 PM

Quote:

Originally Posted by unSpawn

Segei, please realize when you post you're supposed to help out people. Having only ten posts here this user may or may not have the amount of knowledge (yet) a pro possesses or might have phrased things differently than a pro would. So do look beyond that and please ditch the pedantic tone of voice.

I'm helping the person pointing out the glaring contradiction in his/her post.

I really do not understand why the OP can't run Perl, Python or any other scripting language on his/her system.

If/when the OP in more details describes the kind and limitation of his/her system, I'll think of ways of overcoming the limitations. I suspect the system is an embedded one, but I have no proof yet. I think I can offer a solution for, say, an ARM based system, why should anyone put the carriage before the horse ?

Sergei Steshenko · 07-13-2010, 04:55 PM

And, after thinking a little bit, I think I also have an "organizational" solution, i.e. there might be a possibility to perform the needed task not by CPU power of the system under question/consideration, but, again, why should anyone put the carriage before the horse ?

salasi · 07-13-2010, 04:58 PM

Quote:

Originally Posted by shakazzolo

This is a script that I use to cleanup the /lib/firmware and /lib/modules but its really slow...

The script has several stages: one possibility is to find out which stage is slow and then think about a way of speeding that up (I'd guess the 'find' part, because it is easy to waste time in find, but that's a guess and information would be better).

Also you might think of describing the operation of the script; I have no idea what other files there are that you are trying not to delete.

David the H. · 07-13-2010, 05:06 PM

The OP stated flat-out from the beginning that perl and python aren't available to him, so focusing on any "contradictions" in what he said is simply unproductive.

Assuming I understand the script correctly (and that's not 100% certain), it looks like it can indeed be made more efficient. It uses a slow, check-all-the-names-individually-against-each-other double-nested loop that's bound to be quite slow.

I've whipped this replacement up, which should be much faster, because it batch processes the operations. However this means it comes at the cost of losing the detailed output that showed which module matched which file. On the other hand, I've made a couple of other modifications you might like.

Code:

#!/bin/bash

$wkdir=${1:-$PWD}  #defaults to current directory if directory isn't specified.

modlist=$(lsmod | awk '($1!="Module"){print $1}')
filelist=$(find $wkdir -type f -print )

matchlist=$(  echo "$filelist" | grep -i -F "$modlist"  )
deletelist=$( echo "$filelist" | grep -iv -F "$modlist" )


#echo "These files were matched:"   #uncomment to display the list of files
#echo "$matchlist"                  #that do not match the module list
#echo

echo "=============================="
echo
echo "These files will be deleted:"  
echo "$deletelist"
echo
echo "confirm? (y/n)"

read answer

case $answer in

     Y*|y*) echo "$deletelist" | xargs #rm -rf  #uncomment the rm here when 
                                                #satisfied that it works.
            echo "All files deleted"
      ;;

     *) echo "operation aborted"
        exit 1
      ;;

esac

exit 0

You can specify the directory you want to run the script on, so you don't actually need to cd to the directory itself.

./delmods.sh /lib/modules

It will first print out a list of files that have no matches, and are due to be removed, then ask you to confirm the operation. The actual removal is done by this command:

echo "$deletelist" | xargs rm -rf

I've commented out the delete command itself until you can be sure it's working correctly.

You can also uncomment the list of files that are excluded, but it means an even longer output, so I didn't include it by default.

In any case, please make sure that it's really doing what you want it to before using it, since I'm not completely sure what the original was doing, and I may have messed something up.

Sergei Steshenko · 07-13-2010, 05:11 PM

Quote:

Originally Posted by David the H.

The OP stated flat-out from the beginning that perl and python aren't available to him, so focusing on any "contradictions" in what he said is simply unproductive.
...

And I do not accept that claim in the first place. Because I know how to build Perl - it is nowadays simple. I've done it a number of times myself.

Again, the easiest solution might be outside the coding plane.

unSpawn · 07-13-2010, 05:14 PM

Quote:

Originally Posted by Sergei Steshenko

I really do not understand why the OP can't run Perl, Python or any other scripting language on his/her system. If/when the OP in more details describes the kind and limitation of his/her system, I'll think of ways of overcoming the limitations. I suspect the system is an embedded one, but I have no proof yet. I think I can offer a solution for, say, an ARM based system, why should anyone put the carriage before the horse ?

A lot of questions can't be answered qualitatively well due to lack of information. Asking for clarity should then be the way to go IMHO.

unSpawn · 07-13-2010, 05:16 PM

Quote:

Originally Posted by shakazzolo

I can only run bash scripts and compiled programs (C/C++) on that system... so python or perl is not an option

As you can see from the replies more information would be welcome. The more verbose you are the potentially better advice can be tailored to suit your particular situation.

David the H. · 07-13-2010, 05:19 PM

By the way, I have to wonder if a script like this is really such a good idea in the first place. All it does is delete everything that doesn't match the modules that are currently loaded into the kernel. But what if there's a module you need that isn't loaded at the time the script is run?

Sergei Steshenko · 07-13-2010, 05:26 PM

Quote:

Originally Posted by David the H.

By the way, I have to wonder if a script like this is really such a good idea in the first place. All it does is delete everything that doesn't match the modules that are currently loaded into the kernel. But what if there's a module you need that isn't loaded at the time the script is run?

- that was fast.

David the H. · 07-13-2010, 05:26 PM

Quote:

Originally Posted by Sergei Steshenko

And I do not accept that claim in the first place. Because I know how to build Perl - it is nowadays simple. I've done it a number of times myself.

And why should you not accept it, when the poster states it up-front? The impression I get is that may not be for technical reasons, but due to some kind of company policy or similar. At the very least you should take him provisionally at his word, and ask for clarification if and when necessary. It's just a simple scripting question, after all, so why make it more complicated than it needs to be?

Sergei Steshenko · 07-13-2010, 05:32 PM

Quote:

Originally Posted by David the H.

And why should you not accept it, when the poster states it up-front? The impression I get is that may not be for technical reasons, but due to some kind of company policy or similar. At the very least you should take him provisionally at his word, and ask for clarification if and when necessary. It's just a simple scripting question, after all, so why make it more complicated than it needs to be?

Because in my life I've heard too many claims about impossibility which in the end happened to be the results of lack of knowledge, imagination, due diligence, negligence, etc.

And I suspect that the issue is not a scripting/coding one, but the one one of overall system architecture, though "system" might be too strong a word here.

ghostdog74 · 07-13-2010, 07:25 PM

Quote:

Originally Posted by shakazzolo

This is a script that I use to cleanup the /lib/firmware and /lib/modules but its really slow... is there any smarter way for doing this?

I can only run bash scripts and compiled programs (C/C++) on that system... so python or perl is not an option

Code:

export mymod=`lsmod | awk -F' ' '{print $1}' | grep -v Module`
for file in `find . -type f`
 do
 testResult=""
 for mod in $mymod
   do
	tmpVar=`echo $file | grep -i $mod`
	if [ -n "$tmpVar" ]; then
		testResult=$mod
		break
	fi
   done

 if [ -n "$testResult" ]; then
	printf "$file is used by $testResult \n"
 else
	printf "Deleting $file\n"
	rm -Rf $file
 fi
done

its slow because you use tools that overlap in function, eg awk and grep. Also, for every file that you find, you create another loop to grep your strings. another approach might be

Code:

lsmod | awk '!/Module/{print $1}'  > template
find . -type f -iname "<specify probable names>" | while read -r FILE
do
   grep -f template "$FILE"
   .....
done

This way you don't need to loop the modules, but make use of grep's -f to do the job.
Also, you can try using grep's -R option to recurse instead of find command.