Perl and concurrency

exscape · 03-22-2008, 04:06 PM

I'm pulling my hair out, and still can't figure a decent way to do this out... What I want to do is to create a program that somehow creates subprocesses (via threads, fork() or whatever), with the ability to have a hash shared between them.
Specifically, I need to share two objects and a hash. The children then needs to be able to set hash keys that points to object instances... I think.

A bit more verbose:
I'm writing a caching DNS server, for the sole purpose of learning. At the moment, it works great as a passthrough (the caching isn't implemented yet, due to this), but only handles one request at a time. If you throw a bunch of slow queries, the last ones handled will be REALLY slow instead of just "slow".

I tried to set up some basic caching (for testing purposes only, since the TTL is ignored etc etc), by saving the Net::DNS::Packet for the answer into a hash, i.e.

Code:

if (defined $cache{$q->string}) {
                print STDERR "Got it cached! NOT sending to upstream server!\n" if DEBUG;
                $answer = $cache{$q->string};
        } 
        else {
                print STDERR "Sending request to upstream server...\n" if DEBUG;
                $answer = $resolver->send($in_req);
                say STDERR " Done" if DEBUG;
                
                $cache{$q->string} = $answer;
        }

That works too, until I try to add the concurrency part... With fork(), I tried using IPC::Shareable, which worked until the hash grew, and the program died with an out-of-semaphores error.
Then I spent quite a lot of time to recompile perl with ithreads, only to run in to the snag that "Thread 1 terminated abnormally: Invalid value for shared scalar at test.pl line 60"
Line 60 is "$cache{$q->string} = $answer", in other words, setting a key in the shared hash to a Net::DNS::Packet reference.

Is there an easy way around this? Heck, how do the other similar programs work?!

jailbait · 03-22-2008, 04:32 PM

I think that you need to add some locking logic to your program. When a variable is shared among more than one thread and any of the threads update the variable then you have to lock on that variable in order to prevent simultaneous write access. I think that you need to add locks for the two objects and the hash which are shared across threads.

http://perl.apache.org/docs/2.0/api/...eadRWLock.html

-------------------
Steve Stites

exscape · 03-22-2008, 04:59 PM

Quote:

Originally Posted by jailbait

I think that you need to add some locking logic to your program. When a variable is shared among more than one thread and any of the threads update the variable then you have to lock on that variable in order to prevent simultaneous write access. I think that you need to add locks for the two objects and the hash which are shared across threads.

http://perl.apache.org/docs/2.0/api/...eadRWLock.html

-------------------
Steve Stites

Locking is probably a good idea, but this seems to be a compile-time failure.
This code also fails, on line 7 (the one and only):

Code:

use strict;
use warnings;
use threads;
use threads::shared;
use Net::DNS::Resolver;

my $res : shared = Net::DNS::Resolver->new;

chrism01 · 03-22-2008, 08:45 PM

In theory that syntax should work, but I always had more success with:

Code:

    # Create shared queues
    # Queue to thread that sends radius pkts to remote radius servers
    $cfg::send_queue = new Thread::Queue;
    threads::shared::share($cfg::send_queue);

ie declare var, then share it immediately.

Typical locking is to use an anonymous block, lock inside block, lock unlocks automatically when it goes out of scope ie exit from block. "Lock late, unlock early", otherwise prog wastes time waiting.
(IOW, do minimal processing in block)

Code:

    {
        lock(%cfg::send_msg_threads);
        $cfg::send_msg_threads{$cfg::tid} = 'R';
    }

HTH

exscape · 03-23-2008, 04:53 AM

Quote:

Originally Posted by chrism01

In theory that syntax should work, but I always had more success with:

Code:

    # Create shared queues
    # Queue to thread that sends radius pkts to remote radius servers
    $cfg::send_queue = new Thread::Queue;
    threads::shared::share($cfg::send_queue);

ie declare var, then share it immediately.

[/code]
HTH

I don't even have to share it before it fails

Code:

use strict;
use warnings;
use threads;
use threads::shared;
use Thread::Queue;
use Net::DNS::Resolver;

my $q = Thread::Queue->new;
$q->enqueue(Net::DNS::Resolver->new);

Fails on the last line (with an error message pointing to Thread/Queue.pm line 90 - someone needs to tell them about Carp

).
The manual for Thread::Queue talks about this, though, and unless somebody rewrites the modules to be thread-safe, seems this is a no go.

chrism01 · 03-24-2008, 08:12 PM

If Net:: DNS::Resolver produces a non-shareable object (as per http://search.cpan.org/~jdhedden/Thr...pm#LIMITATIONS), then simply create a shared hash or array to store the refs in and then enqueue the hash key value/array element num instead.
I had to do something similar, because when I started with thrs, queues only supported scalars for enqueue.