How to "reverse direction" of POSIX message queue in Linux?

d3f · 10-22-2021, 12:34 PM

I am trying to figure out how to "reverse" a single POSIX message queue to enable two-way communication between a client and server process while only using one POSIX message queue (i.e. only one memory segment at a time). The purpose of this is to cut memory overhead in half for IPC with many clients. This is, in essence, a half-duplex setup where two-way communication is supported, but messages can only flow in one direction at a time.

The standard approach is to use one queue for client -> server and one queue for server -> client as shown below (this is basically my current code).

Client:

Code:

...
int read = mq_open("servertoclient", O_RDWR | O_CREAT, 0600, 0);
int write = mq_open("clienttoserver", O_RDWR | O_CREAT, 0600, 0);
char send_buffer[8192];
mq_send(write, send_buffer, 8192, 0); //send message to server
char receive_buffer[8192];
mq_receive(read, receive_buffer, 8192, 0); //receive response from server
mq_close(write);
mq_close(read);
mq_unlink("servertoclient");
mq_unlink("clienttoserver");
...

Server:

Code:

...
int write = mq_open("servertoclient", O_RDWR | O_CREAT, 0600, 0);
int read = mq_open("clienttoserver", O_RDWR | O_CREAT, 0600, 0);
char send_buffer[8192];
mq_send(write, send_buffer, 8192, 0); //send message to client
char receive_buffer[8192];
mq_receive(read, receive_buffer, 8192, 0); //receive response from client
mq_close(write);
mq_close(read);
mq_unlink("servertoclient");
mq_unlink("clienttoserver");
...

I am looking for a way to accomplish almost the same thing, but only using a single message queue at a time, not one for each direction simultaneously. The only difference is that with the single queue, simultaneous sending/receiving will not be supported, which is OK in my situation. Server and client will use some kind of special 'code' that signals a queue reversal. This is akin to a radio where you talk, then release the button and the receiver's radio beeps, letting them know you've finished talking. Then you listen to the receiver's message until your radio beeps. But the sender and receiver can never talk over each other.

Something along these lines:

Client:

Code:

open a single queue with mq_open for writing to the server
send some data
send a special message notifying the server that the queue is to be reversed
prepare queue for reading (not sure how to do this)
read data from server until the 'reverse' message is received, then revert queue to write
... keep going like this until a terminating message is received or client exits

unlink the queue

Server:

Code:

open a single queue with mq_open for reading from client
read in data
if the terminating message is read, 
   prepare queue for writing to client
send data and finish by sending the special 'reverse' message
prepare queue for reading
... keep going like this until a terminating message is received or server exits

unlink the queue

The client will always initiate the first message, so there is never any danger of the server wanting to send a message initially, for example.

I am not open to using any other method of interprocess communication (such as shared memory, sysv queues, pipes, socket tcp/udp, etc.).

wpeckham · 10-23-2021, 10:07 AM

What you are describing is very like a communication channel. Currently you have one that is asynchronous, what you WANT is one that is synchronous. You may need a handshake symbol or signal to release ownership of the channel, but that is easy enough to do.

I seem to recall doing that in assembler for RS-232 communications about half a century ago. The code may not apply, but the concepts certainly should.

dugan · 10-23-2021, 10:58 AM

Well, you can simply destroy the queue and re-establish a new one (read at one end, write at the other, or vice versa) in response to an "over" message...

But honestly, your requirement makes less and less sense to me each time I think about it. You already have a full-duplex message queue that supports messages going in both directions. That means it already supports a protocol where the application layer (that would be your C code) knows when to send and when to receive, and signals that (to the other application) by sending an "over" message. You do not need the message queues themselves to enforce those restrictions.

Is your actual technical problem trying to determine when the applications should send and when they should receive?

d3f · 10-23-2021, 01:26 PM

Quote:

Originally Posted by dugan

Well, you can simply destroy the queue and re-establish a new one (read at one end, write at the other, or vice versa) in response to an "over" message...

But honestly, your requirement makes less and less sense to me each time I think about it. You already have a full-duplex message queue that supports messages going in both directions. That means it already supports a protocol where the application layer (that would be your C code) knows when to send and when to receive, and signals that (to the other application) by sending an "over" message. You do not need the message queues themselves to enforce those restrictions.

Is your actual technical problem trying to determine when the applications should send and when they should receive?

Yes, since posting this I have discovered that the message queue already supports two-way communication by default and that there is no need to 'reverse' the direction. I had initially been under the impression that queues were like pipes and that data could only flow in one direction. You are right, my current problem is how to synchronize the two processes across a single queue.

I am working with something along these lines (by the way, I'm using C++ not C):

Code:

CommInterface class:

private:
string queue_name;
int queue_descriptor;

void onMessageReceive(){ //called when other process sends this process a message

char buffer[8192];
mq_receive(buffer);
//do something with message

}

void sendMessage(char* message){

mq_send(message);
?? somehow use mq_notify to let the other process know that this process just send a message 

}


CommInterface::CommInterface(string name){

queue_descriptor = mq_open(name converted to char);
queue_name = name;

?? somehow set up onMessageReceive to be called asynchronously whenever the other process sends a message to the queue

}

CommInterface::~CommInterface(){
mq_close(queue_descriptor);
mq_unlink(queue_name.c_str());

}

I read the entire linux manual over message queues and found something called mq_notify, but I cannot seem to implement it properly. What is the simplest way to set up a class member function to be called async with mq_notify?

Or should I not be using mq_notify at all?