read() hanging when receiving stream of data on serial tty

Rusttree · 01-08-2017, 11:25 PM

I'm running into a weird problem trying to read a stream of data on an ARM iMX6 processor using a serial tty port. The data is coming from an external device that outputs 100kbps in 3ms bursts. It blasts data out for 3ms, then goes idle for 3ms, then blasts data, goes idle, etc.

The problem is that read() blocks for about a second to read even just a single byte. Obviously, if several bytes of data are arriving every 3ms, it shouldn't be a whole second before a byte is available to read. I also noticed that the number of bytes in the buffer jumps from 0 to 4095 when I monitor the buffer using ioctl and FIONREAD. read() won't unblock until the buffer gets to 4095, at which point I can read all of the bytes correctly.

Here's some of the troubleshooting and thoughts I've had so far:

1. 4095 appears to be the size of the RX buffer in the serial driver (imx.c).

2. At 100kbps, 1 parity bit, 2 stop bits, and the data streaming with a 50% duty cycle (on for 3ms, off for 3ms), I calculate it takes just about 1 second to fill the 4095 byte buffer. I think this is the cause of the blocking I'm observing.

3. If I disable blocking on the serial port using O_NONBLOCK, read() just returns -1 until the buffer gets to 4095.

4. I think what's happening is the data is arriving too fast and the serial driver's RX interrupt is getting called constantly. The driver never gets a chance to do anything else besides service the interrupt. With this in mind, I tried disabling RX just before doing read() and then re-enabling (by flip-flopping the CREAD bit). I figured that would temporarily stop the interrupt from being called. That kind of worked, but only for a short time. After several seconds, the buffer would suddenly jump back to 4095 bytes and lock up. No further reading or flushing of the buffer had any effect. Here's the code snippet:

Code:

        
  while(1)
  {
      terminal.c_cflag &= ~CREAD ;     //Turn off RX
      ioctl(fd,TCSETS2,&terminal);
      ioctl(fd,FIONREAD,&n_rx_bytes);  //Get the number of bytes in buffer
      printf("n: %d\n",n_rx_bytes);    //print number of bytes to screen

      for(i=0;i<n_rx_bytes;i++)
      {
          read(fd,v+i,1);
      }
 
      terminal.c_cflag |= CREAD ;  //Turn RX back on
      ioctl(fd,TCSETS2,&terminal); 
      ... //Do other stuff
  }

Does anyone have some troubleshooting advice to try next? I can't change anything about the external device as its a black-box commercial product.

suicidaleggroll · 01-10-2017, 11:19 AM

I haven't seen that before...how are you initializing the serial port? It sounds like it might be a bug in the UART kernel module, have you brought it up to the distro maintainer?

There's no way 115200 baud (I'm assuming that's what you meant) async serial is too fast for an iMX6.

jpollard · 01-10-2017, 12:33 PM

It really depends on the line discipline being used.

By default a serial line will wait until there is a newline... This is done to allow the "user" to delete bad characters and make corrections before committing the line for processing.

The other thing that happens by default is echo... The output gets sent under the assumption that the remote is "ready to receive".

One thing you can do is to set raw mode - see manpage on stty basically:

Code:

raw    same as -ignbrk -brkint -ignpar -parmrk  -inpck  -istrip  -inlcr
       -igncr  -icrnl  -ixon  -ixoff -icanon -opost -isig -iuclc -ixany
       -imaxbel -xcase min 1 time 0

And you can set/clear these via the ioctl you already have (though you might want to use hardware flow control to prevent data over-run http://www.tldp.org/HOWTO/Text-Terminal-HOWTO-11.html)

BTW, data over run of serial lines can happen even at low speeds as other activity on the system may delay getting back to the process to read the data. Serial lines tend not to have that large a hardware buffer (16 bytes was the largest when I was coding serial lines - it may be larger now).

Rusttree · 01-11-2017, 01:10 AM

Quote:

Originally Posted by suicidaleggroll

There's no way 115200 baud (I'm assuming that's what you meant) async serial is too fast for an iMX6.

It actually really is 100kbps. Atypical, but that's what this particular device outputs. It also uses inverted logic (idle low, start bit high), even parity, and 2 stop bits. So I think atypical is what they were going for.

Here's how I set up the serial port:

Code:

fd = open ("/dev/ttymxc2", O_RDONLY | O_NOCTTY );
ioctl(fd, TCGETS2, &terminal);
terminal.c_cflag |= (CLOCAL | CREAD) ;
terminal.c_cflag |= PARENB ; //enable parity
terminal.c_cflag &= ~PARODD ; //even parity
terminal.c_cflag |= CSTOPB ;  //2 stop bits
terminal.c_cflag &= ~CSIZE ;
terminal.c_cflag |= CS8 ;
terminal.c_lflag &= ~(ICANON | IEXTEN | ECHO | ECHOE | ISIG) ;
terminal.c_oflag &= ~OPOST ;
terminal.c_cflag &= ~CBAUD;
terminal.c_cflag |= BOTHER;
terminal.c_ispeed = 100000;
terminal.c_ospeed = 100000;
ioctl(fd, TCSETS2, &terminal);

@jpollard, the external device does not use flow control of any kind and I cannot make it do so. I'm in non-canonical mode, so new lines and echos aren't the issue.

After digging around online, I found a post by someone on another forum describing a similar issue. They discovered it had something to do with the UART DMA. Apparently if there isn't enough idle time between bytes, the buffer is never serviced and the user can't access the data. It's not until all 4096 bytes goes through before it gives up and finally relinquishes the buffer to the user. When that person disabled UART DMA in their device tree, the problem went away. Unfortunately the device tree I'm using doesn't seem to have the same option to disable UART DMA.

Edit: Here's the post from the other website I found: https://www.toradex.com/community/qu...er-issues.html

In the end, I've "solved" the problem by forgoing the UART driver completely. My application is very simple and has no blocking code (by design), so I wrote a function modelled after devmem2 to access the registers directly in my code. I come from a microcontroller background, so manipulating the UART registers directly is very natural to me. Now I can grab the serial data as soon as it arrives without any software-induced delay.

I'd still love to get to the bottom of what that supposed DMA hangup is, but for now I can move on.

Doug G · 01-11-2017, 07:56 PM

Quote:

I'd still love to get to the bottom of what that supposed DMA hangup is, but for now I can move on.

I haven't done much work with linux serial ports but lots of serial devices through multiplexers on various systems. One guess, perhaps xon-xoff flow control was enabled and had a problem, such as your end sending an xoff to prevent buffer overruns and the other end doesn't ever respond to it. I've seen computer to computer serial connections, serial printers, and other devices hang because of flow control problems.

suicidaleggroll · 01-11-2017, 09:15 PM

Quote:

Originally Posted by Rusttree

It actually really is 100kbps. Atypical, but that's what this particular device outputs. It also uses inverted logic (idle low, start bit high), even parity, and 2 stop bits. So I think atypical is what they were going for.

Weird...

Quote:

Originally Posted by Rusttree

Here's how I set up the serial port:

Code:

fd = open ("/dev/ttymxc2", O_RDONLY | O_NOCTTY );
ioctl(fd, TCGETS2, &terminal);
terminal.c_cflag |= (CLOCAL | CREAD) ;
terminal.c_cflag |= PARENB ; //enable parity
terminal.c_cflag &= ~PARODD ; //even parity
terminal.c_cflag |= CSTOPB ;  //2 stop bits
terminal.c_cflag &= ~CSIZE ;
terminal.c_cflag |= CS8 ;
terminal.c_lflag &= ~(ICANON | IEXTEN | ECHO | ECHOE | ISIG) ;
terminal.c_oflag &= ~OPOST ;
terminal.c_cflag &= ~CBAUD;
terminal.c_cflag |= BOTHER;
terminal.c_ispeed = 100000;
terminal.c_ospeed = 100000;
ioctl(fd, TCSETS2, &terminal);

@jpollard, the external device does not use flow control of any kind and I cannot make it do so. I'm in non-canonical mode, so new lines and echos aren't the issue.

I don't believe jpollard was referring to flow control, but rather the Linux serial driver's tendency to internally buffer all incoming data until a "newline" (0x0A) is found. I see you haven't set raw mode in your serial port setup. Is the incoming data ASCII or binary?

FWIW - this is my typical serial port setup for binary data (115200 8N1):

Code:

struct termios options;

  *sfd = open(device, O_RDWR | O_NOCTTY | O_NDELAY);
  if (*sfd == -1)
  {
    fprintf(stderr, "unable to open %s\n",device);
    exit(1);
  }
  else
  {
    fcntl(*sfd, F_SETFL, FNDELAY);
  }

  tcgetattr(*sfd, &options);

  cfsetispeed(&options, B115200);
  cfsetospeed(&options, B115200);

  cfmakeraw(&options);

  options.c_cflag &= ~(PARENB | CSTOPB | CSIZE | CRTSCTS);
  options.c_cflag |= (CLOCAL | CREAD | CS8);

  options.c_oflag &= ~OPOST;
  options.c_cc[VMIN] = 1;
  options.c_cc[VTIME] = 0;

  tcsetattr(*sfd, TCSANOW, &options);