Serial communication in real-time process delayed by lower-priority processes (Linux)

Question

I have a real-time process sending occasional communication over RS232 to a high speed camera. I have several other real-time processes occupying a lot of CPU time, doing image processing on several GPU boards using CUDA. Normally the serial communication is very fast, with a message and response taking about 50 ms every time. However, when the background processes are busy doing image processing, the serial communication slows way down, often taking multiple seconds (sometimes more than 10 seconds).

In summary, during serial communication, Process A is delayed if Process B, C, etc., are very busy, even though process A has the highest priority:

• Process A (real-time, highest priority): occasional serial communication
• Processes B, C, D, etc. (real-time, lower priority): heavy CPU and GPU processing

When I change the background processes to be SCHED_OTHER (non-real-time) processes, the serial communication is fast; however, this isn't a solution for me, because the background processes need to be real-time processes (when they are not, the GPU processing doesn't keep up adequately with the high speed camera).

Apparently the serial communication is relying on some non-real-time process in the system, which is being pre-empted by my real-time background processes. I think if I knew which process was being used for serial communication, I could increase its priority and solve the problem. Does anyone know whether serial communication relies on any particular process running on the system?

I'm running RHEL 6.5, with the standard kernel (not PREEMPT_RT). It has dual 6-core CPUs.

At Erki A's suggestion, I captured an strace. Apparently it is a select() system call which is slow (the "set roi2" is the command to the camera, and the "Ok!" at the end is the response from the camera):

write(9, "set roi2"..., 26)             = 26 <0.001106>
ioctl(9, TCSBRK, 0x1)                   = 0 <0.000263>
select(10, [9], NULL, NULL, {2, 0})     = 1 (in [9], left {0, 0}) <2.252840>
read(9, "Ok!\r\n", 4096)                = 5 <0.000092>

The slow select() makes it seem like the camera itself is slow to respond. However, I know that isn't true, because of how the speed is impacted by changing the background process priorities. Is select() in this case dependent on a certain other process running?

If I skip the select() and just do the read(), the read() system call is the slow one.

Answer 1

Depending on your serial device/driver, the serial communications are most likely relying on a kernel worker thread (kworker) to shift the incoming serial data from the interrupt service routine buffers to the line discipline buffers. You could increase the priority of the kernel worker thread, however, worker threads process the shared work queue. So increasing the priority of the worker thread will increase the priority of the serial processing along with a whole bunch of other stuff that possibly doesn't need the priority boost.

You could modify the serial driver to use a dedicated high priority work queue rather than a shared one. Another option would be to use a tasklet, however, both these require driver level modifications.

I suspect the most straight forward solution would be to set the com port to low latency mode, either from the command line via the setserial command:

setserial /dev/ttySxx low_latency

or programatically:

struct serial_struct serinfo;
fd = open ("/dev/ttySxx");
ioctl (fd, TIOCGSERIAL, &serinfo);
serinfo.flags |= ASYNC_LOW_LATENCY;
ioctl (fd, TIOCSSERIAL, &serinfo);
close (fd);

This will cause the serial port interrupt handler to transfer the incoming data to the line discipline immediately rather than deferring the transfer by adding it to a work queue. In this mode, when you call read() from your application, you will avoid the possibility of the read() call sleeping, which it would otherwise do, if there is work in the work queue to flush. This sleep is probably the cause of your intermittent delays.

Answer 2

You can use strace to see where it locks up. If it is more than 10 seconds, it should be easy to see.