Can in-depth Linux serial programming fix this?

CoderMan · 10-21-2009, 05:05 PM

Hi. I've posted about certain aspects of this problem before, but I'm at something of a cross-roads and would appreciate some advice from some people who have a background in Linux serial programming.

The situation: I've got a work project involving communication between a Linux PC and a Parallax boe-bot over bluetooth. The connection is (emulated?) serial communication via rfcomm, so on the PC side I communicate with a /dev/rfcomm* character file. The communication is asynchronous.

The problem: The boe-bot has a single tasking processor and no hardware serial input buffer. Consequently, the boe-bot cannot receive/send data and do other things at the same time. In fact, I only have about a 20 millisecond i/o window and it comes at irregular intervals. But I need the robot to be constantly sending/receiving data and doing other things at the same time.

In the past what I have done to send data is to just repeatedly send the data and hope that the robot gets it at least once. Aside from being very inefficient, and losing a lot of characters, this also makes it hard to receive data back from the robot. I am not a real serial programmer, but I tried to write a very simple C program that uses putc and getc to catch a "signal" from the robot (READY TO RECEIVE) and then sends the data quickly, but for some reason the data is lost. (Not sent fast enough?)

My question: Is what I am trying to do even feasible? If I invest a lot of time learning about termios.h and fcntl.h and ioctl.h and what not will I be able to make this work? Or can I do this without getting in that deep? Or is this just not workable without a robot that has a hardware serial input buffer?

JohnGraham · 10-21-2009, 05:54 PM

I don't know anything about bluetooth, but if it's emulating a serial port there may be some way to use hardware flow control, which is exactly what you want - with a CTS/RTS line pair, you'll tell the robot you're ready to send data, and it'll tell you when it's ready to receive, and vice-versa.

I don't know how (or if) that's implemented in rfcomm, but it's definitely worth looking into.

CoderMan · 10-22-2009, 05:33 PM

Quote:

Originally Posted by JohnGraham

I don't know anything about bluetooth, but if it's emulating a serial port there may be some way to use hardware flow control, which is exactly what you want - with a CTS/RTS line pair, you'll tell the robot you're ready to send data, and it'll tell you when it's ready to receive, and vice-versa.

I don't know how (or if) that's implemented in rfcomm, but it's definitely worth looking into.

Hmm... I implemented Hardware Flow Control before, but it didn't fix the problem. From the BSTAMP programming manual:

Quote:

When you design an application that requires serial communication between BASIC Stamp modules, you have to work within these limitations:
• When the BASIC Stamp is sending or receiving data, it can’t execute other instructions.
• When the BASIC Stamp is executing other instructions, it can’t send or receive data. The BASIC Stamp does not have a serial buffer as there is in PCs. At most serial rates, the BASIC Stamp cannot receive data via SERIN, process it, and execute another SERIN in time to catch the next chunk of data, unless there are significant
pauses between data transmissions. These limitations can sometimes be addressed by using flow control; the Fpin option for SERIN and SEROUT (at baud rates of up to the limit shown in Table 5.94). Through Fpin, SERIN can tell a BASIC Stamp sender when it is ready to receive data. (For that matter, Fpin flow control follows the rules of other serial handshaking schemes, but most computers other than the BASIC Stamp cannot start and stop serial transmission on a byte-by-byte basis. That’s why this discussion is limited to communication between BASIC Stamp modules.)

So according to this, the problem is that the PC won't start/stop communication on a single-byte's notice, and so data will still be lost or sent at the wrong time. So I guess this circles back to my original question: Is there a way to tell Linux or the PC hardware to handle this the way we want?

theNbomr · 10-23-2009, 12:24 AM

Yup. Send all of your data character-at-time, and use some kind of timer to pace the process to a rate that your microcontroller can handle. select() sounds like it will be useful here.
--- rod.

Wim Sturkenboom · 10-23-2009, 01:46 AM

No, in my opinion it will not help. If HW flowcontrol does not work, you can consider software flowcontrol. Your bstamp will, at the beginning of the 20ms window, indicate that it can receive data (XON) after which the PC will start sending data. Once the bstamp can no longer receive data (at the end of the 20ms window), it will indicate this (XOFF) and the PC will stop sending.

Please note that these are the basics. If the PC was not on when the bstamp send the XON, the PC will never now that it can send. So you have to implement something that the bstamp will resend the XON if it has not received any data in e.g. a minute.
There are de-facto values for XON and XOFF, but they might interfere with your data.

Regardless if you can implement XON/XOFF, a decent protocol/commandset needs to be defined/implemented; sending and hoping will not work (as you already figured out). When the bstamp receives data, it needs to acknowledge so the PC knows. If no acknowledge is received by the PC (within a certain time), it will resend. The acknowledge can indicate OK, errors and carry data.

If you haven't thought about a protocol, some ideas
command:
<cmd><msglen><optional data0>..<optional dataN><crc>
acknowledge:
<OK><msglen><optional data0>..<optional dataN><crc>
<error><msglen><optional data0>..<optional dataN><crc>

Hope this helps. Below a start link for the software flow control

http://en.wikipedia.org/wiki/Software_flow_control

theNbomr · 10-23-2009, 10:03 AM

If your window of opportunity is only 20ms long, then using software flow control would seem to waste about half of it (assuming 9600 bps). This is really a synchronization issue, so using hardware flow control should make sense from the perspective of use of the precious time resource. However, hardware flow control is not exposed to userspace applications.
It seems that the microcontroller needs to be the timing master, so it could send a short stream of timed bytes that both ends can use as a time synchronizer. After that, some messaging can occur, where each end can send/receive messages that are sync'd in time. After the completion of a message dialog, the next dialog could be re-sync'ed using the special preamble message from the microcontroller. In order to create appropriate delays between characters sent to the microcontroller, you need to work around the built-in efficiencies provided by the OS driver, which tries to send a complete message buffer as quickly as possible, most likely driven by interrupts from the UART. Sending a message containing a single byte, followed by a programmed delay is the only way to get around that mechanism, at least from a userspace application. This is one place where the OS gets in the way, and a DOS style application would probably be much easier to craft, at least from the standpoint of the serial communications.
I surmise that the BASIC Stamp is not equipped with any kind of UART, and must therefore use a bit-bang method of producing serial data. This would explain its inability to send and receive asynchronously. As a further complication, the rfcomm driver may not even have any concept of hardware flow control, since no such thing exists in RF communications. This problem may be difficult to solve in a purely userspace application, and scrutiny of the existing driver is probably going to be a requirement to understand its behavior.
It isn't clear to me exactly what the structure of the complete communications channel looks like. I am guessing that the BASIC Stamp is attached to some kind of Bluetooth 'dongle' that is used to translate the serial data to RF, and the PC end uses an on-board Bluetooth driver supported by the rfcomm driver? If so, there will be unknown propagation delays in the dongle as it buffers data in each direction, and without being able to control the data timing in that device, no amount of fiddling with timing or other synchronization effort will ever work reliably. Your strategy of sending data redundantly may be as good as it gets.
As I read this, I see that I have evolved my original optimistic stance to a more pessimistic view. I hope I have at least exposed some issues for contemplation and study.
--- rod.

jlinkels · 10-24-2009, 08:30 AM

I have looked at the Boe-bot's web page, and beside the Basic stamp I could not find any indications of a Bluetooth channel on these devices. So it this a customized version? If so, how exactly is Bluetooth implemented, and how is communications between Bluetooth and the Basic stamp?

If the Basic stamp is not able to communicate while it is doing other things, it this as seen from the application program. There must be some primitive operating system on board to run the Basic interpreter. Is communication on that level also impossible? As it seems, this device has a I2S interface, what does it do, it that not available? What is the interface of the Bluetooth device in the Boe-bot? Since the device ssems to be modified already, is it not possible to paste a dual ported hardware buffer between the Bluetooth and the Basic stamp? A piece of programmable logic would do job. I know this is 1995 technology but that is when I moved away from hardware development.

The problem as you describe, a 20 ms window on random(?) moments makes it very hard to communicate, and it might even be impossible because of the synchronization act.

jlinkels

CoderMan · 10-27-2009, 04:36 PM

Quote:

Originally Posted by jlinkels

I have looked at the Boe-bot's web page, and beside the Basic stamp I could not find any indications of a Bluetooth channel on these devices. So it this a customized version? If so, how exactly is Bluetooth implemented, and how is communications between Bluetooth and the Basic stamp?

If the Basic stamp is not able to communicate while it is doing other things, it this as seen from the application program. There must be some primitive operating system on board to run the Basic interpreter. Is communication on that level also impossible? As it seems, this device has a I2S interface, what does it do, it that not available? What is the interface of the Bluetooth device in the Boe-bot? Since the device ssems to be modified already, is it not possible to paste a dual ported hardware buffer between the Bluetooth and the Basic stamp? A piece of programmable logic would do job. I know this is 1995 technology but that is when I moved away from hardware development.

I'm not precisely sure, but I believe that the I2S interface is for generating audio tones. (The Basic Stamp has an instruction that generates a tone of a specified frequency.)

As far as pasting in a dual ported hardware buffer: How hard/expensive do you think it would be to build such a device? I'm not an engineer (more of an applications programmer) but I might be able to get the help of an EE student if I knew what it was I was asking of him. I believe all serial input is routed through one pin and all serial output is routed through one pin, so I guess just two lines of data would need to be buffered.

BTW, a previous poster asked what BT device is on the chip. The answer is that there isn't one: The bluetooth signals are received by an eb500 (Embedded Blue) module that fits into a slot on the "Board of Education". The Bstamp doesn't know anything about Bluetooth: the programmer uses the same commands (SERIN, SEROUT) as are used for normal serial communication but sends the data to different pins so the data is picked up by the eb500 instead of the routed through the serial port.

theNbomr · 10-27-2009, 07:10 PM

Wouldn't it just be cleaner and easier to use something with a little more horsepower than a BASIC Stamp? I know these things cost very little, but if you start kludging on custom hardware, the expense goes way up, and it is still a kludge. There are some pretty small boards that draw very little current, run Linux, have all sorts of on-board IO, and can be had for a little over $100; probably less if you are lucky enough to find a really good match. You might even find one that has Bluetooth on-board, and avoid the eb500, too. ARM based single board computers tend to fill this niche.
--- rod.

jlinkels · 10-27-2009, 08:25 PM

I have checked quickly on the eb500 and didn't see anything which points into the direction of this irregular 20 ms window.

Now it is alway easy for someone not deeply involved to say such things, and I only looked quickly at the eb500, so I am talking and asking mainly out of ignorance.

As far as I could see in the eb500 application notes, it talks to the Bstamp on the serial level, quite transparently of any other running tasks. This is exactly what I expect, the underlying (primitive) operating system takes care of this communication (probably using interrupt handlers) and makes the I/O visible and available to the Basic user program running in the Bstamp. In now way the user application should handle this I/O, just do the read and writes on a Basic level.

However you don't seem someone who didn't study this Bstamp thing in-depth, so I ought to be careful with my assumptions.

I have the pdf datasheet of the eb500, can you point me to a paragraph where the behavior and the 20 ms window is described? Or is it in some other datasheet of the Boe? Or is it because of your application? I assume it is not possible to reprogram the Basic stamp? (Not the Basic of course, but the underlying OS?)

If indeed you are right about this 20 ms thing, the design of the whole thing might be flawed. theNBomr might be right in suggesting a whole new chip.

My previous solution would use a piece of programmable logic (in the 1990-ies they were called PAL's) and implement a serial receiver in it. Edge detector, shift register, buffer. This receiver would receive everything from the eb500 and raise a signal when its buffers are full. Then, when the Bstamp issues the proper signal, it would pump all this data into the Bstamp, probably at a higher rate than 9600 bps. To accomplish this an EE would be helpful, the PAL chip costs next to nothing (the S&H is more expensive) but you need to dedicate quite some time to program this PAL. But still, you are trying to compensate for a flawed design if this is really what you need.

jlinkels

CoderMan · 10-27-2009, 09:44 PM

First of all, I'd like to thank everyone who has responded so far. I don't think I've ever got this much help in a post before.

Quote:

Originally Posted by jlinkels

I have checked quickly on the eb500 and didn't see anything which points into the direction of this irregular 20 ms window.

You are correct, there is no inherent 20 ms window in bstamp/eb500 communications. The 20 ms window is imposed by me. You see, I want my boe-bot robot to be constantly multi-tasking. That is, I want the robot to be constantly sending/receiving data while it is continuously moving around, picking things up, and so forth. Unfortunately, however, the Basic Stamp processor is single-tasking and has no hardware interrupts, meaning that it can only be doing one thing at a time.

For parallax motors to move, they must receive a pulse from the processor approximately once every 20 ms. Consequently, to achieve continuous data input/output and continuous robot activity, I can have at most a 20 ms serial input/output timeout before the processor must stop i/o and move on to dealing with moving the motors and other activites.

Sorry, I would have explained that earlier, but I hate to flood people with information until I'm sure they want it.

Quote:

If indeed you are right about this 20 ms thing, the design of the whole thing might be flawed. theNBomr might be right in suggesting a whole new chip.

Yeah, this equipment is not great for this kind of application, but unfortunately it is the equipment I inherited. I actually put in a funding request for some LEGO NXT Robots -- hopefully that will pan out down the road.

Quote:

My previous solution would use a piece of programmable logic (in the 1990-ies they were called PAL's) and implement a serial receiver in it. Edge detector, shift register, buffer. This receiver would receive everything from the eb500 and raise a signal when its buffers are full. Then, when the Bstamp issues the proper signal, it would pump all this data into the Bstamp, probably at a higher rate than 9600 bps. To accomplish this an EE would be helpful, the PAL chip costs next to nothing (the S&H is more expensive) but you need to dedicate quite some time to program this PAL. But still, you are trying to compensate for a flawed design if this is really what you need.

Thanks for the info.

For the curious, here is an update on my activities so far:

I went ahead and started playing around with the more in-depth linux serial programming, using read(), write(), termios structs, and all that fancy stuff. I'm quite the n00b there, but I was able to create some test code: The robot communicates a 'ready to receive' character (my arbitrary character, not software-fc) and then waits for input with a 20 ms timeout. If it receives a response before the timeout, then it echos back the same character that was sent to it.

My C code on the PC side receives all input, and when it detects the 'ready to receive' character it sends a confirmation character immediately.

I did some statistical testing with this code, and it seems to show that for every 120 confirmation characters sent out by the PC, roughly 80 of those characters are received by the robot and echoed back.

Here is the C code:

Code:

#include <stdio.h>   /* Standard input/output definitions */
#include <string.h>  /* String function definitions */
#include <unistd.h>  /* UNIX standard function definitions */
#include <fcntl.h>   /* File control definitions */
#include <errno.h>   /* Error number definitions */
#include <termios.h> /* POSIX terminal control definitions */

/*
 * 'open_port()' - Open serial port 1.
 *
 * Returns the file descriptor on success or -1 on error.
 */

int open_port(void)
{
  int fd; /* File descriptor for the port */
  
  fd = open("/dev/rfcomm0", O_RDWR | O_NOCTTY | O_NDELAY);
  if (fd == -1)
    {
      /*
       * Could not open the port.
       */
      
      perror("open_port: Unable to open /dev/rfcomm0 - ");
    }
  else
    fcntl(fd, F_SETFL, FNDELAY);
  
  return fd;
}

int main()
{
  char c = 0;
  int fd = open_port();
  if(fd == -1)
    return -1;

  struct termios options;
    
  // set 9600 Baud

  cfsetispeed(&options, B9600);
  cfsetospeed(&options, B9600);

  // Whatever this does...

  options.c_cflag |= (CLOCAL | CREAD);

  // 8N1

  options.c_cflag &= ~PARENB;
  options.c_cflag &= ~CSTOPB;
  options.c_cflag &= ~CSIZE;
  options.c_cflag |= CS8;

  // Raw input
  options.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);

  // No software flow-control
  options.c_iflag &= ~(IXON | IXOFF | IXANY);

  tcsetattr(fd, TCSANOW, &options);
  int d_count = 0;
  int q_count = 0;
  int ii = 0;
  while(ii < 200)
    {
      ssize_t c_read = read(fd, &c, 1);
      if(c_read > 0)
	{
	  //	  printf("%c\n", c);
	  if(c == 'D')
	    {
	      write(fd, "Q", 1);
	      d_count++;
	      ii++;
	    }
	  else if(c == 'Q')
	    {
	      q_count++;
	      ii++;
	    }
	}
    }
  printf("%d %d\n", q_count, d_count);
  close(fd);
}

'D' is the 'ready-to-receive' character and 'Q' is the confirmation character.

Wim Sturkenboom · 10-27-2009, 11:34 PM

Found serial buffer ic.

Quote:

Protean Logic Inc.'s RSB509B is an easy to use, low cost IC that buffers RS232 serial data for a host microprocessor. The RSB509B stores up to 32 bytes of serial data received on the input pin and retransmits data to the host through the interface pin. The RSB509B frees the host microprocessor from serial data timing concerns. The real time considerations of RS232 interrupt handling are thus eliminated or reduced. Processors that emulate RS232 hardware via software, like the FBasic TICkit or the PARALLAX Basic STAMP, can perform other tasks while the RSB509B buffers serial data. The processor can retrieve the stored data when it has the time to spare by pulsing the interface pin. When using a TICkit or a STAMP as the host processor, the user program simply outputs a high on the interface pin immediately before the rs_receive or serin line.

The page also has some sample code for the basic stamp.

If it can be integrated in your design is up to you; I was too lazy to research the eb500

Found a price of $8 on http://www.proteanlogic.com/product_periph_rs232.shtml (I assume US$).

PS 1) google search term was serial buffer; did not expect much but it was the first hit
PS 2) be aware of baudrate limitation

jlinkels · 10-28-2009, 10:01 AM

Quote:

Originally Posted by Wim Sturkenboom

Protean Logic Inc.'s RSB509B is an easy to use, low cost IC that buffers RS232 serial data for a host microprocessor.

OK, apparently development did not stop in 1995. Very good find.

jlinkels