LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-08-2009, 07:51 PM   #1
matth45
LQ Newbie
 
Registered: Mar 2008
Posts: 4

Rep: Reputation: 0
worker thread queue (or thread pool) in C under linux?


I'm porting some code I wrote that uses the Win32 api to linux. Is there a worker thread queue api or library in linux? I'd like to find something similar to the Win32 QueueUserWorkItem, since I'd rather not roll my own thread queue.

There's an intel project under GPL called Threading Building Blocks, but it's a C++ library. I also found libcprops, but the project doesn't seem to be maintained and doesn't actually implement a queue, just a pool. Finally there are kernel workqueues (linux/workqueue.h), but these seem to be made with kernel space driver code in mind. Would they work in a userspace program?

Can anyone point me to a library or api that will let me schedule jobs to be run by a pool of threads with a fixed size?

Thanks!
 
Old 11-08-2009, 08:43 PM   #2
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Hi -

It sounds like you might be looking for something a little higher-level, but I'd encourage you to consider pthreads:

http://www.yolinux.com/TUTORIALS/Lin...ixThreads.html
http://www.ibm.com/developerworks/li.../l-posix1.html

It also sounds like you're *not* considering C++ ... but I'd look at "Boost" in preference to Intel's TBB:

http://en.wikipedia.org/wiki/Boost_C%2B%2B_Libraries

IMHO .. PSM
 
Old 11-08-2009, 08:50 PM   #3
Seedhom
LQ Newbie
 
Registered: Oct 2009
Location: Vestal NY
Distribution: CentOS
Posts: 3

Rep: Reputation: 0
Worker Thread pool for Linux

Quote:
Originally Posted by matth45 View Post

Can anyone point me to a library or api that will let me schedule jobs to be run by a pool of threads with a fixed size?

Thanks!
AFAIK there are no such equivalent set of APIs that are de facto in the linux world. I am not familiar with the projects you referenced, but depending on your needs, implementing pthread pool may not be complex undertaking IMHO. At the other end of the spectrum, there is the open-source resource manager Simple Linux Utility for Resource Management (SLURM) which has a plug-in job scheduler if that is what you are looking for.
 
Old 11-08-2009, 11:48 PM   #4
matth45
LQ Newbie
 
Registered: Mar 2008
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks for the links, both of you. I'm somewhat familiar with both pthreads and Boost. My impression was that they would both be decent starting places if I wanted to implement a thread pool myself, but this is exactly what I'm trying to avoid. SLURM seems way too heavy.

The program I'm porting has a section where it runs some functions that traverse a 3D dataset where the columns and planes can alternatively be processed independently. The idea of the thread pool was to amortize the creation cost of threads created to process the rows/cols across many jobs. I assume you guys know that's what I was getting at by your answers, but I suppose a little clarification never hurts. Anyway, I'm looking for speed here, since this happens at video rates, on a single system. (I have a system with many cores, large dataset, etc). I'm no threading expert, so I could see myself easily spending a week creating my own pthreads work queue and debugging it. Do you know any example code for something like this? It seems hard to believe that no one is doing this in C in linux. Or is it easier than I'm thinking?
 
Old 11-09-2009, 12:39 AM   #5
matth45
LQ Newbie
 
Registered: Mar 2008
Posts: 4

Original Poster
Rep: Reputation: 0
http://docs.sun.com/app/docs/doc/816-5137/ggedn?a=view

Here's one example.
 
Old 04-13-2010, 02:20 PM   #6
dundee
LQ Newbie
 
Registered: Apr 2010
Posts: 2

Rep: Reputation: 0
Quote:
Originally Posted by matth45 View Post
Here's one example.
But be careful, it's example for Solaris, not for Linux.
Some parts therefore won't work, for example this causes segfault:

Code:
pthread_create(NULL, attr, void *(*func)(void *), void *arg);
 
Old 04-13-2010, 03:03 PM   #7
dundee
LQ Newbie
 
Registered: Apr 2010
Posts: 2

Rep: Reputation: 0
Quote:
Originally Posted by matth45 View Post
I'm porting some code I wrote that uses the Win32 api to linux. Is there a worker thread queue api or library in linux? I'd like to find something similar to the Win32 QueueUserWorkItem, since I'd rather not roll my own thread queue.

There's an intel project under GPL called Threading Building Blocks, but it's a C++ library. I also found libcprops, but the project doesn't seem to be maintained and doesn't actually implement a queue, just a pool. Finally there are kernel workqueues (linux/workqueue.h), but these seem to be made with kernel space driver code in mind. Would they work in a userspace program?

Can anyone point me to a library or api that will let me schedule jobs to be run by a pool of threads with a fixed size?

Thanks!
Maybe this link can help you: http://people.clarkson.edu/~jmatthew.../pthreadsPool/
It is a reusable pool of general-purpose threads library.
 
Old 03-13-2012, 07:19 AM   #8
Pithikos
LQ Newbie
 
Registered: Mar 2012
Posts: 2

Rep: Reputation: Disabled
Check this implementation out

You can check my implementation of a thread pool on github or sourceforge. What makes it more helpful than the previous posts is that it has full documentation of design and usage.
 
  


Reply

Tags
api, pool, port, queue, win32


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
python thread safety: printing from thread to redirected stdout - safe? BrianK Programming 2 10-11-2010 11:28 AM
the return value of getpid() called from main thread and new thread r identical !!! cryincold Programming 3 02-29-2008 01:37 AM
Main thread sending notification to child thread rajesh_b Programming 1 09-22-2004 09:15 AM
configure qt thread issue (just compiled qt w/ -thread option) cleff Linux - Software 8 05-07-2004 11:11 PM


All times are GMT -5. The time now is 07:16 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration