How to implement a queuing system?

aleet2600 · 05-12-2004, 03:32 PM

Anyone knows if there is any open source out there that implements a queuing system, kinda like a print spooler?

What I want to do:

I have say a program, foobar.

I want to implement a web front end for user to submit jobs to the queue.

The Web front end will collect a few parameters from user, then put the parameters into the queue.

The queue will read the jobs w/ parameters then call foobar with the parameters.

User can look at the jobs in the queue.

User can remove jobs from the queue.

Anyone knows anything like this out there? Or Perl module that will do that?

david_ross · 05-12-2004, 03:38 PM

Why not just store the queued items in a database table or even a flat file?

aleet2600 · 05-12-2004, 04:10 PM

Hi David,

If I store it in a DB/File, I assume you want to setup cron job every minute to kick off foobar, then foobar will grab the list of jobs and execute them.

This won't work in my situation because the reason I am trying to implement the queue is to avoid waiting. I want the jobs to be done as them come in.

Thanks for your advise. Please let me know if you have some more.

david_ross · 05-12-2004, 04:18 PM

In this case you will most likely want to write a script/program that will constantly listen for connections to either a file or TCP port.

Most languages are capable of that and there are serveral IO modules for perl which will help.

aleet2600 · 05-12-2004, 04:32 PM

I see. So I can write a server that listen for jobs/query/remove coming from Web interface, and maintain the queue inside the server. I assume that's RPC? Any lead on that if I write it in Perl?

Thanks!

deiussum · 05-12-2004, 04:46 PM

It sounds like maybe you want some sort of message queue system. IBM has one called MQ that comes with WebSphere. I don't know the details of it, but I have looked at it briefly. It may be a bit "full-featured" for your needs, though.

There is also a POSIX message queue supported by gcc that I had used at one time, but I don't remember the specifics of it. Do a google search on "Linux message queue" and you will likely find some stuff...

I'll see if I can find some of the example code I had written when looking at that stuff for you.

Edit:

I found a simple test program that uses the message queue stuff. It does it all in one app, but the send/recv stuff could be done in separate apps. I haven't looked at this code in years, so I can't guarantee it's the most clean...

Code:

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>
#include <sys/stat.h>
#include <errno.h>
#include <iostream>

// Message buffer struct
struct msgbuff_test1
{
    long mtype;     // Type of message
    char mtext[1024];   // You can specify the size you need
};

const int MSGTYPE_TEST1 = 1;    // ID for this message type

int g_qid;

using namespace std;

int main()
{
    int rval;

    // Create/get the queue
    g_qid = msgget(0, S_IRWXU);

    if (g_qid == -1)
    {
        cerr << "msgget() failed - " << errno << endl;
        return -1;
    }
    else
        cout << "MsgQueue created" << endl;

    // Put a message on the queue
    msgbuff_test1 s;
    s.mtype=MSGTYPE_TEST1;
    strcpy(s.mtext, "test");

    rval = msgsnd(g_qid, &s, sizeof(msgbuff_test1), 0);

    if (rval == -1)
    {
        cerr << "Error putting message on the queue - " << errno << endl;
        return -1;
    }
    else
        cout << "Message placed on the queue" << endl;
    // Get a message off the queue
    msgbuff_test1 r;
    rval = msgrcv(g_qid, &r, sizeof(msgbuff_test1), MSGTYPE_TEST1, 0);

    if (rval == -1)
    {
        cerr << "Error getting message from the queue - " << errno << endl;
    }
    else
        cout << "Message retrieved from the queue - " << r.mtext << endl;
    return 0;
}

aleet2600 · 05-12-2004, 05:16 PM

deiussum,

Yeah it might be a little over kill for me.

I am planning to use Named Pipe so I can coordinate/protect the read/write at the queuing program. And it will sleep until getting SIG.

As a amature programmer, now the challenge is how to write it in Perl.

Thanks!

paulsm4 · 05-12-2004, 10:29 PM

Hi -

Interesting question. I wouldn't give up on the database idea (just yet), and the odds are pretty good that you can do most (perhaps all) of it in Perl.

1. You've got (at least) two guys:
a) The web interface for monitoring and writing to the queue
b) Some "worker agent" who processes items in the queue

2. Let's assume you implement the queue with a database:

The web side becomes very simple: you can write it as a Perl
CGI (probably using the Perl CGI module to make your life easy).

Your "database" consists of two tables: one for the "work items", the
other holds the "head pointer" and the "tail pointer".

If you use the Perl DBI module, the database logic is a piece of cake.

If your database supports transactions (begin tran/commit/rollback), then
you've also solved any potential concurrency problems (at least on the web side)

3. If your database supports triggers, and if the trigger can launch a system
command, then you've also solved the "worker agent" part of the equation: you
just process each new command as it comes in, and the system does the queueing
for you.

MS-SQL Server, for example, meets all of these criteria.

Oh yeah - and you can easily open a security hole the size of the Grand Canyon
if you're not careful ;-)!

4. But let's say that instead you wanted to write the "worker agent" in Perl, too.

OK - it would probably be a separate process.

5. The easy thing would be for it to simply poll the database for any new
jobs to process.

6. Alternatively, you could create a message queue with the IPC:SysV module.

You would *not* use the message queue to communicate the job details
(remember - that's in the database).

Instead, you would have the "worker agent" *block* on the message queue
whenever it becomes empty. It will then sleep until somebody (i.e. your web
CGI) sends a "wakeup signal". At which point the "worker agent" wakes up
and (using the Perl DBI module) reads all of the work items (we know there's
at least one, and potentially mode) in the queue.

The only advantage of the message queue (or any other "signal" you might
devise) is that it saves you the overhead of polling.

'Hope that helps .. PSM

kooch · 05-12-2004, 10:35 PM

A named pipe sounds like the best idea. A database dependent system seems like overkill.

paulsm4 · 05-12-2004, 10:45 PM

That's true if you're just putting work items in and taking them out.

But you'll recall that another requirement was the ability to monitor (and perhaps administer) what's *in* the queue.

Throw concurrency issues into the mix, add the fact that Perl:

BI is Good (and that the author wants to write in Perl, if at all possible) ... and suddenly databases don't look so unattractive anymore.

IHMO ...

kooch · 05-12-2004, 11:03 PM

I didn't realize the OP wanted to monitor/modify the contents of the queue, shows what I get for not reading the entire thread.

Listen to paulsm4 and disregard my previous post (=