Reading serial data in realtime in Python

Question

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.

The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)

Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)

On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)

The problem:

• When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
• When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.

Here is the code (I omitted most part of the code, but the loop is the same):

ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
  data_raw = ser.readline()
  print(data_raw)

Anyone knows why pySerial takes so much time to read from the serial port till the end of the line? Any help?

I want to have this in real time.

Thank you

Answer 1

You can use inWaiting() to get the amount of bytes available at the input queue.

Then you can use read() to read the bytes, something like that:

While True:
    bytesToRead = ser.inWaiting()
    ser.read(bytesToRead)

Why not to use readline() at this case from Docs:

Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.

You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

Answer 2

You need to set the timeout to "None" when you open the serial port:

ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False) 

This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end: line = ser.readline()

Once you have the data, it will return ASAP.

Answer 3

A very good solution to this can be found here:

Here's a class that serves as a wrapper to a pyserial object. It allows you to read lines without 100% CPU. It does not contain any timeout logic. If a timeout occurs, self.s.read(i) returns an empty string and you might want to throw an exception to indicate the timeout.

It is also supposed to be fast according to the author:

The code below gives me 790 kB/sec while replacing the code with pyserial's readline method gives me just 170kB/sec.

class ReadLine:
    def __init__(self, s):
        self.buf = bytearray()
        self.s = s

    def readline(self):
        i = self.buf.find(b"\n")
        if i >= 0:
            r = self.buf[:i+1]
            self.buf = self.buf[i+1:]
            return r
        while True:
            i = max(1, min(2048, self.s.in_waiting))
            data = self.s.read(i)
            i = data.find(b"\n")
            if i >= 0:
                r = self.buf + data[:i+1]
                self.buf[0:] = data[i+1:]
                return r
            else:
                self.buf.extend(data)

ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)

while True:

    print(rl.readline())