Serial Communication with more than 2 devices
 

Hi all,

In my application I have to interface with more than 6 external interfaces using serial ports. Lets consider 3 of them say A,B,C. And B sends data in a special way where every packet of B is a single line of the form
$abcde,f,g,h,i,j*hh<CR><LF> where hh is the checksum.
Port configuration for B is

The BAUDRATE2 etc are macros for actual stuff. I m using RHEL4,Qt4.2.1 and C++. Also, I read 100 bytes at a time into a char buffer and then check for packet validity according to interface rules.
When I run A & B together, everything is fine. But when C is introduced after 3-4 hrs I receive frames from B such that every character of
frame is followed by a newline and after some more time only $ is received as the only character followed by a newline. But when I close the application window and open it again (without closing the parent window) it runs fine. This leads me to believe that may be there is something wrong with the port setting. Because when I add CRTSCTS to subsystem C and run A,B,C together then B goes down after 10-15 minutes itself. Can somebody point out as to where is the problem?

Pretty much just spitballing, here, but I would be looking at the driver(s) for your 6 serial ports. I don't know of a standard architecture that provides for 6 serial ports on a PC (not that you've specified that this is a PC architecture; is it?), so I'm guessing that there is one or more custom drivers for the serial interfaces. There is probably some interrupt sharing involved, and this sounds like the likely place where data from different channels would get interleaved. Without more detail, this is about as much as I would speculate.

How are you reading the devices? You say you read 100 byte blocks, but are the newline delimited frames always exactly 100 bytes in length? How do you open the character device for reading/writing? Are you using blocking reads for each port? If yes, how are you managing to get around the loop to read all ports while one channel is blocked? You have configured the port for canonical input, which should delimits reads by newlines, but you say you read 100 bytes at a time, which seems to be conflicting purposes.

It sounds like the devices are sending unsolicited data. Can you connect a known-good terminal emulator (I recommend C-Kermit) to each port, and let it run to test the ability of the serial driver(s) to manage the data? This will tend to separate your code from the driver code as a source of error.
--- rod.

Thanks for the reply rod
 

Thanks for the reply rod.

I would be more specific now. Firstly I am only testing 4 interfaces because by default Redhat/Fedora supports 4 serial ports. For additional serial ports one has to recompile the kernel with support enabled for extra serial ports.
Secondly, the interface I am having problem with sends data at the rate of 5 Hz. ie 5 frames per second.
Thirdly, I am using interrupt driven methodology whereby I use SIGIO and sigaction to define my own signal handler whose primary aim is to wake up the individual threads for processing. Until then they are waiting on a wait condition. I have used a single signal handler which when invoked checks on the function descriptors to wake up the necessary thread for processing.
Fourthly, I am using non-blocking reads (otherwise the system will hang).
Fifthly, frame size is 29 bytes including CR,LF.
Now can you suggest something rod

In your signal handler, how do you distinguish which serial interface caused the signal, in order to wake up the appropriate thread? Why do you use canonical input processing?

I still recommend using some known-good serial terminal emulator to perform some testing. Do that and see what results.

--- rod.

hi rod,

I have got over the problem of broken frames now. What I did was reduce the buffer size for collecting data from 100 bytes to 30 bytes( one more than the bytes in the frame I receive). Also I added my own ResetPort function which is called when Wind goes down due to some reason. It does nothing more than close and open port again. One of your suggestions worked mate.
I use canonical processing only for one interface which sends frames delimited by CR,LF (line oriented input). Is it wrong?
Now a second more prodding problem remains. When 2 threads are running they run fine but when a third is introduced then the application hangs in the newly introduced thread after about 7-8 hours. I know that without putting out the actual code here no one can say anything. It is extremely hard to debug multithreaded programs, I used valgrind and helgrind but there messages are hard to understand. Can you suggest some other free tools? One more thing the newly introduced threads do both send and receive of data simultaneously.

As far as your query. I already told you that I compare the file descriptor to identify the thread which has received data. The SIGIO signal is generated when I/O is available at serial port. I use a siginfo struct which gives all the information about the signal i.e the function descriptor etc. using siginfo fd I compare it with fd of serial port when I opened it. If they match "bingo". I do this comparsion for all threads. These comparisons are clubbed in a InputSignal function which is a signal handler when SIGIO is generated. I hope it is clear to you now.

Canonical input processing would be fine if you were trying to read data line-at-a-time fashion. If you are reading in larger blocks, it doesn't make sense to me. Your method of associating SIGIO to the appropriate handler thread seems fine. How are you able to determine that the third thread is the one that is hanging? Since threads are all still part of one process, and one thread that hangs causes the whole process to hang, might it be prudent to use multiple processes? I don't know how much inter-process communication would be required for your application, but it may make your overall application more robust, and possibly more debuggable.
When sending and receiving data concurrently, it is possible to have interrupts overlap. That is, when an interrupt to service a non-empty receive buffer occurs, it may coincide with the case of an empty transmit buffer. My main experience with interrupt driven serial IO is at the assembler level, where it is necessary to test for the two conditions, and service both if necessary on a single interrupt. This may not apply at the level you are working at, as the OS may generate separate signals for each event. Failing to handle this circumstance will cause communications to stall, especially where you are using a query-reply protocol, where responses from the serial device requires a query. Is your device sending unsolicited data, or are you sending queries to cause it to reply? Do you have a timeout mechanism to recover from lost/corrupted queries and replies?
--- rod.

hi rod,

My conclusion was based on the fact that the two thread A,B run correctly together but when C is introduced, system hangs after 5-6 hours of execution. Thus it leads me to believe that thread C is the culprit. Peculiar thing about C thread is that it conducts two way (full duplex) communication with the external system. It first reads the serial port for data and depending upon the data gives a response by writing to the serial port an acknowledgment frame. This happens every 1 second. In case I receive 5 consecutive incorrect frames I show the system as disconnected. The data on the GUI is updated every 1 second by checking a boolean variable which is set by the thread when it received a frame. This technique is followed for every thread in the application. There are two threads which do both send and receive data. In a way you can say that they follow a query reply protocol and I have used timeouts wherever necessary to invoke appropriate behaviour. The problem is manifestation of the hang takes 6 hours and it is difficult to pinpoint as to why this happens. As of now I am screwed.

--lucky

Have you considered using "select" instead of signals? I was doing that for 2 serial ports and found it far less error prone than relying on signals.

Have you declared variables used in the signal handler as "volatile", if they are shared by more than one thread?