Greetings Everyone!

I am currently running Linux 7.2 and I am having some problems in reading and writing the serial ports with "binary" data. I have properly set up the port for "non-canonical input" and "raw output" data. (As was previous, I was set up for ASCII data only which was changed appropriately)

unsigned char XYStatusRequest[4];

XYStatusRequest[0] = 0xB3; // this is just a request buffer to obtain data
XYStatusRequest[1] = 0x58; // from some hardware that I interface to
XYStatusRequest[2] = 0x59;
XYStatusRequest[3] = 0x0D;

Then, I write the data out of serial port #3 as follows....

c = write(fd_serial_port_3, (void *) &XYStatusRequest, sizeof(long));

My debug printf statement always reads 4 bytes written!

This code appears to be working for the writing of the binary data out of the serial port to the hardware device.

Then, the readback function expects 12 bytes of binary data from serial port #3. This code reads back the data from serial port #3 after a delay of 10-20 ms after the write.

unsigned char XYStatusRequestReadback[12];

c = read(fd_serial_port_3, (void *) &XYStatusRequestReadback,
sizeof(XYStatusRequestReadback));

My debug printf statement here reads back 8, 9, or 12 bytes of data.....when I read back 8 or 9 bytes of data, the data is always wrong....when I read back 12 bytes of data, the data is mostly correct, but eventually wrong!

So, THIS code FAILS to retrieve the binary data after numerous iterations from commands at a 1/2 second rate from the host comptuer. Many iterations before this have successful READS as well as WRITES!!!

Any assistance that you can afford to me would be greatly appreciated! Thank you very much for all of your time and consideration!

Eric.

Hi esi-eric,
How do you wait these 10 or 20 ms? Are you using a timer? May be you are receiving the characters before the read command is ready to read it. I suggest you to use the select(2) command to wait for the characters to arrive. Read the man page carefully, because you have to setup some structures, use some predetermined macros, but with these routines you will get all characters in time. I had a problem like this several years ago. If you need some more help, please, describe what speed you use to communicate, how did you set your serial (termios(3)).
Have fun!

Greetings and thank you so very much for the great response!!! I have tried what you have suggested, but the select call
is not timing out anymore, but I am still receiving 8, 9, or 12 bytes of data
back when I peform the read after the select call says I have data to be
read. I am posting below how I set up the serial port and the code which reads the serial port so you can see the whole picture. By
the way, I was using a timer!!! You were absolutely correct!!! And, it was giving me intermittent results. Also, the baudrate that
I am operating at is B38400. (I can't seem to copy and paste from EMACS to the Forum page!!!! What am I doing wrong??)

Anyway, I will just have to type everything out for you....


void CalcXYZStatus()
{
fd_set rfds;
struct timeval timev;
int retrval;

unsigned char XYStatusRequest[4];

XYStatusRequest[0] = 0xB3; // this is just a request buffer to obtain data
XYStatusRequest[1] = 0x58; // from some hardware that I interface to
XYStatusRequest[2] = 0x59;
XYStatusRequest[3] = 0x0D;

c = write(fd_serial_port_3, (void *) &XYStatusRequest, sizeof(long));

unsigned char XYStatusRequestReadback[12];

FD_ZERO(&rfds);
FD_SET(fd_serial_port_3, &rfds);

timev.tv_sec = 5;
timev.tv_usec = 0;

retrval = select(/*1*/(fd_serial_port_3+1), &rfds, NULL, NULL, &timev);

if (retrval) {
c = read(fd_serial_port_3, (void *) &XYStatusRequestReadback,
sizeof(XYStatusRequestReadback));
}

}

void configure_serial_port_3)
{
struct terminios options_serial_port_3;

fd_serial_port_3 = open(MODEMDEVICE_SERIAL_PORT_3, O_RDWR | O_NOCTTY | O_NONBLOCK)

.
. save the old options for the port here...
.
options_serial_port_3.c_cflag |= CLOCAL | CREAD;
options_serial_port_3.c_cflag |= CS8;
options_serial_port_3.c_cflag |= BAUDRATE_CONTROLLER;

options_serial_port_3.c_cflag &= ~CRTSCTS;
options_serial_port_3.c_iflag = IGNPAR | ICRNL;
options_serial_port_3.c_oflag = 0;
options_serial_port_3.c_lflag &= ~ICANON;

options_serial_port_3.c_cc[VMIN] = 1;
options_serial_port_3.c_cc[VTIME] = 0;

.
.
.
I do then a tcflush on the fd
then, I set the new options for the port...

}

thank you so very much once again, and I look forward to your reply. Please let me know if there is any additional information that I can supply you with that can help you assist me.

Eric.

Hi eric,
The following routine was first published in Unix World 1992. Later I adapted it to work with Linux, as the structures involved are slightly different. I use it to communicate with a journal printer:There are some small differences between your and mine as the ECHO, VMIN, etc.
The open command I use is the following:The read routine must take into consideration that may be your device didn't send all the characters when you read the buffer. So, you have to create a read routine which tries to fill the buffer, restarting the process until you get all the characters. As your device always send 12 characters, if you read 8, you must restart the select to expect the last 4. Then, if you read one more, restart it to read the last 3 and so on.
With respect to the wrong readings, I have a question: all the characters are wrong or you can recognize some as an expected part of the message? Two things come to my mind. I don't know the content you expect but,
1 - as you didn't turn off XON, some of the characters may be kept by the xon/xoff feature of the serial line (the same as CTRL/q CTRL/s);
2 - as your routine expect at least 1 character, if you read 9 it is enough to complete the read. Your read routine finish while your device is sending the last 3. Now you send another "request buffer", your device starts to send another answer but you already have 3 in the buffer, creating a wrong result.
I hope it helps. The setting I showed here are to expect 0 characters and to read O_NDELAY. If there are no bytes available, the read doesn't hang.
By the way, to paste the code in this window, I use an xterm see where I cat the file. The xterm can slide for "copy".

Hi osvaldomarques!!!

I have put in your new "open" command with the appropriate settings that you have specified upon
performing the open to obtain a file descriptor. Thank you so very much!

fd_serial_port_3 = open(MODEM_PORT_3, O_RDWR | O_NOCTTY | O_NDELAY | O_EXCL));

I also re-coded my serial port settings to match the ones that you specified as well. Thank you very much! I am using the following....

tcgetattr(fdesc, &tsaved); /* save current settings */
tnew = tsaved; /* initialize new settings */
tnew.c_iflag &= ~IXON; /* reset XON/XOFF */
tnew.c_oflag &= ~OPOST; /* reset post processing */
tnew.c_cflag |= CLOCAL; /* disable modem control signals */
tnew.c_cflag |= CSTOPB; /* set two stop bits */
tnew.c_lflag &= ~ICANON; /* turn off canonical processing */
tnew.c_lflag &= ~ISIG; /* turn off signals */
tnew.c_lflag &= ~ECHO; /* turn off echoing */
tnew.c_cc[VMIN] = 0; /* return when a character read */
tnew.c_cc[VTIME] = 0; /* no time limit set */
tcsetattr(fdesc, TCSANOW, &tnew); /* establish new setting */


The most improvement came when I did as you specified in the following: "The read routine must take into consideration that may be your device didn't send all the characters when you read the buffer. So, you have to create a read routine which tries to fill the buffer, restarting the process until you get all the characters. As your device always send 12 characters, if you read 8, you must restart the select to expect the last 4. Then, if you read one more, restart it to read the last 3 and so on". I have performed this and I am now obtaining better results, but I am still reading back some bad data as I will show you below....I am now always getting some combination yielding 12 characters read back in code which performs successive selects and reads until I get all of the data!!!!! Thank you!

With respect to the wrong readings, I have a question: all the characters are wrong or you can recognize some as an expected part of the message?

Here is an example of bad data....(READBACK)

byte[0] = 0xB3 --> this is Correct as it is the header!
byte[1] = 0x58 ---> this is Correct as well as it is the header!
byte[2] = 0x72 ----> this is WRONG!
byte[3] = 0x0A ----> this is WRONG!
byte[4] = 0x00 ----->BYTES 4 thru 10 are correct!
byte[5] = 0x00 ---------> Correct
byte[6] = 0x00 ---------> Correct
byte[7] = 0x00 ---------> Correct
byte[8] = 0x08 ---------> Correct
byte[9] = 0x09 ---------> Correct
byte[10] = 0x01--------> Correct
byte[11] = 0xE9 -------> this is the checksum and it is WRONG!

So, bytes 2, 3, and 11 are wrong!!!!

Here is what it should have been!!!!

byte[0] = 0xB3
byte[1] = 0x58
byte[2] = 0xEE
byte[3] = 0x0F
byte[4] = 0x00
byte[5] = 0x00
byte[6] = 0x00
byte[7] = 0x00
byte[8] = 0x08
byte[9] = 0x09
byte[10] = 0x01
byte[11] = 0x67

Add up bytes 1 thru 10 to get the proper checksum in byte #11 (0x167 but we drop the "1" from the response
back from the hardware!)

I am very perplexed as to the reason for this bad data. We have a really good Visual Basic program talking to the
same hardware and performing the same calculations on the results read back from our hardware board. Since the
VB program can do it, then why can't I consistently do it in Linux without achieving a "bad data" result after several
iterations at a 1/2 second interval between subsequent writes and reads from this hardware. It is not as if I always fail to receive good data, but STILL alot of the time I am receiving bad data! Based on what I have said, in
this post, if you have any additional suggestions they would be greatly appreciated by me. Thank you very much once again!

By the way, I still don't understand how you use an "xterm" to copy and paste into this window. Would you mind
explaining it again for me please? I have tried today and I was unsuccessful. Thank you so very much for your
continued excellent support of me in this extremely hard issue to solve for me.

Eric.

Hi eric,
Forgive me by an stupid question but, are you sure the checksum of the wrong message is 0xE9 or could it be a transcript error and the original value is 0xE6?
I need to be sure of this value!
About the xterm, at least in kde, when you left click the mouse and drag you copy the text to a transfer area. As the xterm has buffer for more than the lines of the terminal, all you need is to cat your file, search for the cat command using the slide bar, left click over the text you want to copy and holding the mouse button, bring it until the end of the text you want to copy. Please, forgive me as I do not have enough and fluent English vocabulary to explain it.

Hi osvaldomarques,

Linux Visual Basic

0xB3 0xB3
0x58 0x58
0xA6 0xA6
0x0A 0x0D (this is where the descrepency occurs)
0x00 0x00
0x00 0x00
0x00 0x00
0x00 0x00
0x08 0x08
0x09 0x09
0x01 0x01
0x1D 0x1D (checksums are the same!!!)

The reason why you see a checksum in the wrong message that does not make sense is
that our hardware is sending the correct checksum but a partially wrong message (I think).
This is at least what I can determine from my troubleshooting today. Normally bytes[2] and
bytes[3] are the ones that are having the problem during readback from what I find today,
but the correct checksum for the correct message (see visual basic above) is reported back
to me from the hardware. Does this answer your question? Thank you so very much for
your continued excellent support osvaldomarques!!! If you have any additional questions,
please don't hesitate to ask me. By the way, the 2nd and 3rd bytes are the LSB and MSB
of a position encoder that reads position for our system and that is where the problem
may be at.

Eric.

Hi Eric,
I am thinking for some time the causes of these errors. Some questions had arisen:
- Are you using the same computer to run linux and VB for the tests?
- If your answer is "yes", could you be stressing your device with requests as with linux/C you are too much faster the the other environment?
- Otherwise, are you using the same device, cable?
- Are always the same bytes in error (I mean the 3rd and 4th) or the error can happen in any of the 10 bytes following the header?
Waiting your reply,
Osvaldo.

Greetings Osvaldo!

- Are you using the same computer to run linux and VB for the tests?

Yes. I am using the same computer to run the Linux and VB for the tests. The Linux is on
another partition of the same hard disk drive. In addition, all of the hardware is the same as well,
as I just reboot into the new environment and change nothing but what OS and program is
querying the hardware hooked up thru the same serial port.

- If your answer is "yes", could you be stressing your device with requests as with
linux/C you are too much faster the the other environment?

Interestingly enough, the stressing of the requests does not matter since I can reproduce
the problem on a single iteration of me "pinging" the hardware from the Linux code. BUT, just
to let you know, we are normally querying the device at a rate of 10 Hz or every 100 ms from
Visual Basic (which does not fail!!!!) and from Linux at a rate of 2 Hz or 500 ms (which does fail!!!)

- Otherwise, are you using the same device, cable?

Yes, we are using the exact same device, cable, serial port and everything. Nothing moves
at all during the tests, i.e. the hardware and software remain static.

- Are always the same bytes in error (I mean the 3rd and 4th) or the error can happen in any
of the 10 bytes following the header?

If I move the position encoder to another location on the hardware that I am communicating
with, I see the problem move down the chain of bytes. For example if I move the position
encoder to another connector on the hardware, then the next set of bytes would be in error,
for example the 5th and the 6th bytes.

Great point! I have seen the 1st 2 header bytes be in error in the past. BUT, since you have
assisted me in modifying the Linux code so that I always receive 12 bytes of data from the
hardware device, I have not seen the header bytes be in error. IF they ever do show an
error I would see the following....

byte[0] = 0x08;
byte[1] = 0x09;
.
.
.

YES! I did notice today that the first 2 bytes did not come out correct! I had
never seen this happen before today, but all of the bytes were entirely
shifted down by 1. For example...

byte[0] = 0x00;
byte[1] = 0xB3;
byte[2] = 0x58;
byte[3] = 0x...
byte[4] = 0x...
byte[5] = 0x00;
byte[6] = 0x00;
byte[7] = 0x00;
byte[8] = 0x00;
byte[9] = 0x08;
byte[10] = 0x09;
byte[11] = 0x01; /* this is where chksum supposed to be but readback */
/* in the incorrect byte position */

Thank you so very much!

Eric.

P.S. Here is what my write and readback code looks like....


void CalcXYZStatus() {

int c;
int k;
int counter;
unsigned long saved_read_counter = 0;
unsigned long current_read_counter = 0;
fd_set rfds;
struct timeval timev;
int retval;
int maxfd;

if (SendXYStatusRequestFlag == TRUE) {

/* zero out the readback array
*/
bzero(XYStatusRequestReadback,sizeof(XYStatusRequestReadback));

XYStatusRequest[0] = 0xB3;
XYStatusRequest[1] = 0x58;
XYStatusRequest[2] = 0x59;
XYStatusRequest[3] = 0x0D;

c = write(fd_serial_port_3,(void *)&XYStatusRequest[0],4);

maxfd = fd_serial_port_3 + 1;

FD_ZERO(&rfds);
FD_SET(fd_serial_port_3, &rfds);

timev.tv_sec = 5;
timev.tv_usec = 0;

c = 0;
counter = 0;

while (counter < 12) {
retval = select(maxfd, &rfds, NULL, NULL, &timev);

if (FD_ISSET(fd_serial_port_3, &rfds)) {
c = read(fd_serial_port_3,
(void *)&XYStatusRequestReadback[counter],12);

counter = counter + c;

}
else {
printf("The retval is %d\n",retval);
}
}
SendXYStatusRequestFlag = FALSE;

calc_xystatus_checksum();

printf("The calc checksum %x and the read checksum is %
x\n",xaxis_checksum,XYStatusRequestReadback[11]);

if ( (XYStatusRequestReadback[0] == 0xB3) &&
(XYStatusRequestReadback[1] == 0x58) &&
(xaxis_checksum == XYStatusRequestReadback[11]) ) {

calc_xaxis_gurley_encoder();
calc_yaxis_gurley_encoder();
calc_azaxis_gurley_encoder();
calc_xsystem_status();
calcYSystemStatus();
calcZSystemStatus();
XMotor1Errors();
XMotor2Errors();
YMotor1Errors();
YMotor2Errors();
ZMotor1Errors();
ZMotor2Errors();
XYEncoderErrors();
ZEncoderErrors();

}
else {
printf("Either 1st 2 bytes not match or checksum bogus...\n");
printf("Byte 0 is %x\n",XYStatusRequestReadback[0]);
printf("Byte 1 is %x\n",XYStatusRequestReadback[1]);
printf("Byte 2 is %x\n",XYStatusRequestReadback[2]);
printf("Byte 3 is %x\n",XYStatusRequestReadback[3]);
printf("Byte 4 is %x\n",XYStatusRequestReadback[4]);
printf("Byte 5 is %x\n",XYStatusRequestReadback[5]);
printf("Byte 6 is %x\n",XYStatusRequestReadback[6]);
printf("Byte 7 is %x\n",XYStatusRequestReadback[7]);
printf("Byte 8 is %x\n",XYStatusRequestReadback[8]);
printf("Byte 9 is %x\n",XYStatusRequestReadback[9]);
printf("Byte 10 is %x\n",XYStatusRequestReadback[10]);
printf("Byte 11 is %x\n",XYStatusRequestReadback[11]);
printf("The calc checksum %x and the read checksum is %x\n",xaxis_checksum,XYStatusRequestReadback[11]);
exit(1);
}
} // end if


Greetings Osvaldo,

I am posting this edit at 3:50 Eastern Standard Time or 4:50 PM your
time in Brazil. I just found out with Procomm in the loop monitoring the
data coming back to me in the Linux program that the "read" routine, or
the way the read is coded, or the way readback is setup in the serial port
is wrong!!!! In procomm, I obtain the exact message back that I should with the exact checksum. In the message readback in Linux, readback (all 12 bytes readback in one shot) it comes back with a 0x0A where there should be a 0x0D and all other bytes are correct?! How can this happen???? It is not the computer, it is not the connecting hardware, it is not Visual Basic, it is a Linux BUG!!!! Either in my code or in Linux!

Thank you very much once again!

Eric.

Hi Osvaldo,

I am posting this edit at 5:18 PM Eastern Standard Time or 6:18 PM
your time in Brazil. I just confirmed that everytime I get a 0x0D from the
hardware (in any given byte position) that I obtain a 0x0A in the Linux readback function. Is Linux confused with CR and LF characters that are supposed to be just binary data characters or is something just wrong with my "read" function or is it the operating system. Thank you very much once again Osvaldo!

Eric.

Hi Eric,
I observed the errors, often are the replacement of 0x0A by 0x0D (LF by CR). Insert in the initialization, after the line with "~IXON" the following lineI also would suggest you to control the size of the read to not have buffer overflow

Greetings Osvaldo!

tnew.c_iflag &= ~INLCR; /* do not translate 0x0A to 0x0D

I have inserted the above line into the serial port initialization code and it still fails! I have the
reverse problem whereas a 0x0D is translated into a 0x0A! Not a 0x0A into a 0x0D...the 0x0D is
correct and the 0x0A is incorrect!

Thank you very much! and p.s. I am still at work debugging this thing!!!!!

Where can I get a list of these control bytes like "INLCR" for my reference?

Thank you very much once again!

Eric.

------------------------------------------------------------------------------------
I have inserted the following line into the code and it works!!!!
tnew.c_iflag &= ~ICRNL; /* do not translate 0x0D to 0x0A */

Do I still need your code as well to not translate 0x0A to 0x0D????

Thank you very much once again!!!!! I think we finally got this working!!!!!

Eric.

Hi Eric,
You can enter You need both flags. They are independent.
Have fun!
Osvaldo.

Osvaldo,

THANK YOU! for the fixes last night.....the code ran the longest without a fault since its
inception!!!!!

To reiterate from the other day, I am still concerned about other
read backs which are erroneous.......Here is what I stated in an earlier
post......I will be trying the newest code this morning and see if it ever
fails in 1 hours time!!!!! I will let you know the news!

"Great point! I have seen the 1st 2 header bytes be in error in the past. BUT, since you have
assisted me in modifying the Linux code so that I always receive 12 bytes of data from the
hardware device, I have not seen the header bytes be in error. IF they ever do show an
error I would see the following...."

byte[0] = 0x08;
byte[1] = 0x09;
.
.
.
.


This almost like it is putting bytes [9] and [10] in [0] and [1].....

FROM THE OTHER DAY....

"YES! I did notice today that the first 2 bytes did not come out correct! I had
never seen this happen before today, but all of the bytes were entirely
shifted down by 1. For example..."

byte[0] = 0x00;
byte[1] = 0xB3;
byte[2] = 0x58;
byte[3] = 0x...
byte[4] = 0x...
byte[5] = 0x00;
byte[6] = 0x00;
byte[7] = 0x00;
byte[8] = 0x00;
byte[9] = 0x08;
byte[10] = 0x09;
byte[11] = 0x01; /* this is where chksum supposed to be but readback */
/* in the incorrect byte position */

Do you think that we will still have a problem with the erroneous data read back in the
shifting of the bytes, etc. (Why would this happen?) Please let me know what you
think Osvaldo!

Thank you so very much!

Eric.

Hi Eric,
Any asynchronous protocol must have some sync. In standard RS-232 we had SOH (start of heading), STX (start of text) and ETX (end of text).
As you have two header characters you should synchronize in them before start counting the remainder text. In this case, it is important to know if the characters which compose the header can happen in the middle of the message. If it doesn't happen you're safe.
One more thing to try is to read all the characters available on input before you send the request. As you don't hold the read when no characters are available, you can put a read loop until there is characters available; then you send the request, supposing there is no more data flux on the input line.
About the synchronization, it would be like thisI tried to use a example language but it is easier for me to write C than English. You can use, modify these source code without any restrictions.
Cheers,
Osvaldo.

Osvaldo,

Thank you so very much for the excellent reply that you gave to me last night!!!!
I am not able to try your source code now as I don't have the hardware that
I communicate with for a few days. I would like to get back in touch with you
to tell you how it all went!!!! For now, thank you so very much for all of the
assistance and support of my efforts to properly readback binary serial data on
Linux COM ports. With your assistance I have been able to re-write my serial ASCII readback
code so that it is faster and efficent and so that I don't have to wait forever on a
signal handler to give me my readback data from the port. Yes, we are
reading/writing 3 serial ports in ASCII and the 4th serial port in binary!!!! Once
again you are a wonderful help to me in this part of the project!!!! Thank you
once again!!

Eric.