LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-22-2008, 04:06 PM   #1
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Rep: Reputation: 15
Perl and concurrency - sharing data


I'm pulling my hair out, and still can't figure a decent way to do this out... What I want to do is to create a program that somehow creates subprocesses (via threads, fork() or whatever), with the ability to have a hash shared between them.
Specifically, I need to share two objects and a hash. The children then needs to be able to set hash keys that points to object instances... I think.

A bit more verbose:
I'm writing a caching DNS server, for the sole purpose of learning. At the moment, it works great as a passthrough (the caching isn't implemented yet, due to this), but only handles one request at a time. If you throw a bunch of slow queries, the last ones handled will be REALLY slow instead of just "slow".

I tried to set up some basic caching (for testing purposes only, since the TTL is ignored etc etc), by saving the Net::DNS::Packet for the answer into a hash, i.e.
Code:
if (defined $cache{$q->string}) {
                print STDERR "Got it cached! NOT sending to upstream server!\n" if DEBUG;
                $answer = $cache{$q->string};
        } 
        else {
                print STDERR "Sending request to upstream server...\n" if DEBUG;
                $answer = $resolver->send($in_req);
                say STDERR " Done" if DEBUG;
                
                $cache{$q->string} = $answer;
        }
That works too, until I try to add the concurrency part... With fork(), I tried using IPC::Shareable, which worked until the hash grew, and the program died with an out-of-semaphores error.
Then I spent quite a lot of time to recompile perl with ithreads, only to run in to the snag that "Thread 1 terminated abnormally: Invalid value for shared scalar at test.pl line 60"
Line 60 is "$cache{$q->string} = $answer", in other words, setting a key in the shared hash to a Net::DNS::Packet reference.

Is there an easy way around this? Heck, how do the other similar programs work?!
 
Old 03-22-2008, 04:32 PM   #2
jailbait
Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Debian Wheezy, Debian Jessie
Posts: 7,469

Rep: Reputation: 154Reputation: 154
I think that you need to add some locking logic to your program. When a variable is shared among more than one thread and any of the threads update the variable then you have to lock on that variable in order to prevent simultaneous write access. I think that you need to add locks for the two objects and the hash which are shared across threads.

http://perl.apache.org/docs/2.0/api/...eadRWLock.html

-------------------
Steve Stites
 
Old 03-22-2008, 04:59 PM   #3
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by jailbait View Post
I think that you need to add some locking logic to your program. When a variable is shared among more than one thread and any of the threads update the variable then you have to lock on that variable in order to prevent simultaneous write access. I think that you need to add locks for the two objects and the hash which are shared across threads.

http://perl.apache.org/docs/2.0/api/...eadRWLock.html

-------------------
Steve Stites
Locking is probably a good idea, but this seems to be a compile-time failure.
This code also fails, on line 7 (the one and only):
Code:
use strict;
use warnings;
use threads;
use threads::shared;
use Net::DNS::Resolver;

my $res : shared = Net::DNS::Resolver->new;
 
Old 03-22-2008, 08:45 PM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,225

Rep: Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021
In theory that syntax should work, but I always had more success with:
Code:
    # Create shared queues
    # Queue to thread that sends radius pkts to remote radius servers
    $cfg::send_queue = new Thread::Queue;
    threads::shared::share($cfg::send_queue);
ie declare var, then share it immediately.

Typical locking is to use an anonymous block, lock inside block, lock unlocks automatically when it goes out of scope ie exit from block. "Lock late, unlock early", otherwise prog wastes time waiting.
(IOW, do minimal processing in block)
Code:
    {
        lock(%cfg::send_msg_threads);
        $cfg::send_msg_threads{$cfg::tid} = 'R';
    }
HTH
 
Old 03-23-2008, 04:53 AM   #5
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by chrism01 View Post
In theory that syntax should work, but I always had more success with:
Code:
    # Create shared queues
    # Queue to thread that sends radius pkts to remote radius servers
    $cfg::send_queue = new Thread::Queue;
    threads::shared::share($cfg::send_queue);
ie declare var, then share it immediately.

[/code]
HTH
I don't even have to share it before it fails
Code:
use strict;
use warnings;
use threads;
use threads::shared;
use Thread::Queue;
use Net::DNS::Resolver;

my $q = Thread::Queue->new;
$q->enqueue(Net::DNS::Resolver->new);
Fails on the last line (with an error message pointing to Thread/Queue.pm line 90 - someone needs to tell them about Carp ).
The manual for Thread::Queue talks about this, though, and unless somebody rewrites the modules to be thread-safe, seems this is a no go.
 
Old 03-24-2008, 08:12 PM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,225

Rep: Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021
If Net:: DNS::Resolver produces a non-shareable object (as per http://search.cpan.org/~jdhedden/Thr...pm#LIMITATIONS), then simply create a shared hash or array to store the refs in and then enqueue the hash key value/array element num instead.
I had to do something similar, because when I started with thrs, queues only supported scalars for enqueue.
 
  


Reply

Tags
ipc, perl, threads


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
program concurrency problem in C expeliarmus Programming 3 09-07-2007 09:05 AM
Java concurrency xeon123 Programming 3 07-09-2007 05:18 AM
Perl: $data[$i]->[$j] problem PB0711 Programming 3 05-09-2006 12:11 AM
Postfix Concurrency Problem redwolf3 Linux - Software 1 04-13-2005 09:54 AM
Sharing data question. gsmonk Linux - Networking 2 02-22-2003 01:44 PM


All times are GMT -5. The time now is 12:38 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration