multithreaded perl question

chr15t0 · 06-19-2006, 09:08 AM

Hi guys,

I'm just playing with perl threads. I want my script to keep spawing threads (but never any more than 5 at a time). Each thread just does the same thing - it counts to 5 with 1 sec delays and then ends.

I'm having problems determining the number of threads running. Thread->list returns all the threads which are not joined or detached, but I am detaching my threads so that I don't have to wait for them to return. How can I keep track of how many are running?

I have tried a number of things - at the moment my code looks like this:

Code:

#!/usr/bin/perl5.8.7

use strict;
use Thread;

for(my $i=0;;$i++){
	my $thr = new Thread \&thr_handler;

	$thr->detach;

	until( Thread->list < 5){ 
		# this does nothing really
		print " ".Thread->list." threads \n";
		sleep 1;
	}
}

sub thr_handler{
	print STDOUT "joining new thread $\n";

	my $count = 0;
	while($count<5){
		sleep 1;
		$count++;
		#print STDOUT " -$threadid- \n";
	}

};

Does anybody know how I might be able to get this to keep on running with only a maximum of 5 threads running at any time?

cheers

christo

spirit receiver · 06-19-2006, 09:50 AM

First, let me quote the Thread module's documentation:

Quote:

For new code the use of the Thread module is discouraged and the direct use of the threads and threads::shared modules is encouraged instead.

But then, have a look at Thread::Semaphore. It will do all the work for you:

Code:

#!/usr/bin/perl

use strict;
use threads;
use Thread::Semaphore;

my $sem = new Thread::Semaphore( 5 );

while( 1 ){
  $sem->down();
  my $thr = threads->create( 'thr_handler' );
  print "New Thread created.\n";
  $thr->detach();
}

sub thr_handler{
  sleep 5;
  print "Thread finished.\n";
  $sem->up();
};

chr15t0 · 06-20-2006, 03:48 AM

Thanks - I was reading this: http://perldoc.perl.org/threads.html which didn't make any mention of the replacement of Thread by threads..

Thanks a lot.

christo

chrism01 · 06-20-2006, 08:33 PM

"There is another older Perl threading flavour called the 5.005 model, unsurprisingly for 5.005 versions of Perl. The old model is known to have problems, deprecated, and will probably be removed around release 5.10. You are strongly encouraged to migrate any existing 5.005 threads code to the new model as soon as possible.

You can see which (or neither) threading flavour you have by running perl -V and looking at the Platform section. If you have useithreads=define you have ithreads, if you have use5005threads=define you have 5.005 threads. If you have neither, you don't have any thread support built in. If you have both, you are in trouble.

The user-level interface to the 5.005 threads was via the Threads class, while ithreads uses the threads class. Note the change in case. "

See this link: http://72.14.203.104/search?q=cache:...en&lr=&strip=1

sundialsvcs · 06-20-2006, 09:40 PM

For what it's worth... it is usually not a very good idea to establish that "a request == a thread." Not in any language.

It is, imho, a much better idea to establish that "a thread == a worker-bee," and that once a particular thread has been created it will be allowed to survive until the program ends, whether it has work to do right now or not.

When the program starts, it creates a fixed number of threads, all of whom immediately start waiting for a request to arrive on a queue. Each thread, upon dequeueing a request, executes it and then goes back to waiting again, ad infinitum.

At any moment in time, there might be a backlog in the to-do queue, but the number of active worker-threads remains constant. Consequently, the expected throughput (requests/sec) and the total maximum system load remains fairly constant. (When an operating system starts thrashing, the results are never pretty, and this technique largely avoids that.)

chrism01 · 06-21-2006, 02:00 AM

As per sundialsvcs, in some cases creating & deleting (lots of) thrs can cause mem leaks, ime.
Don't forget that (officially) Perl thrs are not prodn ready, although I've found they are fine if used with care. iirc, it's because not all of Perl internals or modules are thr safe (yet).

chr15t0 · 06-21-2006, 03:47 AM

Quote:

Originally Posted by sundialsvcs

For what it's worth... it is usually not a very good idea to establish that "a request == a thread." Not in any language.

It is, imho, a much better idea to establish that "a thread == a worker-bee," and that once a particular thread has been created it will be allowed to survive until the program ends, whether it has work to do right now or not.

When the program starts, it creates a fixed number of threads, all of whom immediately start waiting for a request to arrive on a queue. Each thread, upon dequeueing a request, executes it and then goes back to waiting again, ad infinitum.

At any moment in time, there might be a backlog in the to-do queue, but the number of active worker-threads remains constant. Consequently, the expected throughput (requests/sec) and the total maximum system load remains fairly constant. (When an operating system starts thrashing, the results are never pretty, and this technique largely avoids that.)

That would seem to make a _lot_ of sense, but I can't imagine how you would go about it.. Sprint Reveiver's original response seemed pretty neat and I have been progressing with that without really thinking about all the context switching overheads and the possible resk of memory leaks starts to put me off.. Are there any neat examples or howto's out there that you know of which demonstrate your concept of keeping threads alive and dispatching work to them? How would that be done with the basic example above?

Thanks guys for your input.

christo

spirit receiver · 06-21-2006, 04:06 AM

It wouldn't make any sense with your example above, since the threads in your example actually don't do anything.
But in principle, have a look at the Thread::Queue module. Your threads could be running in an infinite loop, where they try to retrieve the information they need for their job from a queue. If the queue is empty, they'll just be pausing.
Now the main program just needs to put that information into the queue where any of the idle threads might pick it up and start working.

chr15t0 · 06-21-2006, 04:28 AM

ok spirit I'll take a look at Thread::Queue - I agree the example above is over simplistic, but I'm just looking at the fundamentals of how this stuff works before I start integrating it into something more meaty - Just to put it into context, I'm thinking of using a threaded set of 'engines' for my dotcut system (www.dotcut.com) - it's just a personal project I build for fun when I'm not at work. At the moment the dotcut engines are just a set of daemonized perl scripts, which only handle on thread of execution each at any time. My aim is to open that up so an engine might handle say 5 or 6 execution contexts at a time, hence beefing up the parsing and indexing which dotcut has to undertake..

christo

chrism01 · 06-23-2006, 02:38 AM

Just in case you haven't seen it, here's the google cache of perlthrtut (perl thread tutorial). Pretty good doc: http://72.14.203.104/search?q=cache:...en&lr=&strip=1

chr15t0 · 06-23-2006, 03:34 AM

Quote:

Originally Posted by chrism01

Just in case you haven't seen it, here's the google cache of perlthrtut (perl thread tutorial). Pretty good doc: http://72.14.203.104/search?q=cache:...en&lr=&strip=1

just out of curiosity, why d'you post google's cache? Esp when this exists: http://perldoc.perl.org/perlthrtut.html ? (not meant in an unfriendly way

)

christo

chrism01 · 06-25-2006, 07:45 PM

I often find perl.org takes a long time to load up, don't know why, so i just grabbed the google version which i already had bookmarked (for the same reason).

chr15t0 · 06-26-2006, 03:41 AM

aye, cool

christo