Perl Script performance
 

Let me preface by saying that I am trying to increase performance of this service also I am a newbie in this area. The service is a way for users to transfer files across an internal network that has about 22,000 users. Originally when this service was made only about 2000 unique users used this service.

I am reworking some custom scripts. I have been monitoring some of the top services with the "top" command, and I identified a perl script that goes through a folder and opens every file with a special extension file. It reads the first line of every file that it opens which contains metainformation. Instead of using a database or xml file or a one stop location for metadata on file, the original creator of the program saves one file that contains information about a second file

So the script that continually comes to the top goes through each metafile opens that file. There are roughly 20k files that go through at any given time. This system is kinda like a mailbox system for larger files since email has a limit of 10MB. Also, we have hundreds - thousands of users that use this system every hour... My question how big of an impact could this script have on overall server performance? Is changing this script worth an performance increase? What way should I go about changing this model to increase the performance?

Not really any good answer for this...we don't know your environment, or what you're seeing. If that script is taking your CPU to 100% for minutes at a time, then yes...it's time for a rewrite. If it bumps slightly for a few seconds, that's a judgment call on your part.

The better question would be "is that perl script causing you any problems in its current configuration?". If it isn't, then that's something you can rework later. If it IS, then yes, it would be a good idea to rework it, but again, we don't know what resources you have available to you.

In my mind, I would personally keep the metadata in a MySQL database, and reference it, but we don't know the context in which the application operates, so it's really hard to give a solid answer.

Sorry, again I'm new to this. Umm we are running RHEL 4... I have no control over any configuration files nor can I access other parts of the server. We have a team that puts multiple applications on shared servers. I can work with any of the code within our directory. This script runs every time a user goes to download or upload a file. The trend seems to be that the more users using the application(during peak times) and the number of files on the server ( usually Wednesday - Friday ) because files are stored until all users retrieve the file or 5 day has passed, makes the application run very slow. I was told to use the top command and monitor throughout the day which processes reoccur during slow times.... And that's what script popped up. It's a very small script, but it reads anywhere between 5k-15k files to read the first line only (to retrieve the metadata on each file that is uploaded) so that it knows which files to show the person when they log on to the application.

probably it can keep the data in the memory and read metadata only when changed (new file arrived)