PySerial loses data

Question

My problem is, that PySerial seems to lose some data packages and I don't know why.

I have two python scripts, the first one reads data from a text file and writes it to a microcontroller, where the data is manipulated. Then, the microcontroller sends the modified data over a different serial port back to the PC. (for clarification: I need two serial ports, because in the final application, the scripts will be running on different PCs. However, for testing purpose, it is easier to run both scripts on one PC)

So basically, my hardware setup looks like:

PC ----(serial port 1)----> microcontroller
   <---(serial port 2)----- 

After writing the data to the microcontroller I am expecting to get a certain amount of data bytes back. If I use a terminal program (like HyperTerminal) to monitor the data which is received, everything looks fine. However, if I try to read the data with a Python script, I only get a fraction of the expected data bytes.

For example:

+--------------------+--------------------+
| with HyperTerminal | with Python script |
+--------------------+--------------------+
| 1:W:00522          | 1:W:00522          |
| 1:W:00532          | 1:W:00532          |
| 1:W:00518          | 1:W:00522          |
| 1:W:00522          | 1:W:00526          |
| 1:W:00522          | 1:W:00514          |
| 1:W:00526          | 1:W:00520          |
| 1:W:00514          | 1:W:00514          |
| 1:W:00520          | 1:W:00522          |
| 1:W:00520          | 1:W:00526          |
| 1:W:00514          | 1:W:00520          |
| 1:W:00516          | 1:W:00526          |
| 1:W:00522          | 1:W:00520          |
| 1:W:00526          | 1:W:00524          |
| 1:W:00520          | 1:W:00526          |
| 1:W:00520          | 1:W:00532          |
| 1:W:00526          | 1:W:00506          |
| 1:W:00522          | 1:W:00520          |
| 1:W:00520          | 1:W:00526          |
| 1:W:00524          | 1:W:00524          |
| 1:W:00522          | 1:W:00526          |
| 1:W:00526          | 1:W:00514          |
| 1:W:00514          | 1:W:00522          |
| 1:W:00532          | 1:W:00520          |
| 1:W:00506          | 1:W:00510          |
| 1:W:00522          | 1:W:00506          |
| 1:W:00520          |                    |
| 1:W:00526          |                    |
| 1:W:00530          |                    |
| 1:W:00524          |                    |
| 1:W:00526          |                    |
| 1:W:00514          |                    |
| 1:W:00514          |                    |
| 1:W:00522          |                    |
| 1:W:00524          |                    |
| 1:W:00520          |                    |
| 1:W:00510          |                    |
| 1:W:00506          |                    |
+--------------------+--------------------+

As you can see, if I try to read from the serial port with my Python script, I am missing some data. Due to the fact, that I get the expected data if I use a terminal program, I assume, that my Python script has an error.

My Python script for sending data to the microcontroller looks like:

import serial
import re
import time

class digiRealTest():
    def __init__(self):
        #configure serial port 
        self.ser = serial.Serial(7, 9600, parity=serial.PARITY_NONE) 


    def main(self):
        filepath = 'C:\\Users\\Bernhard\\Desktop\\TomatoView\\Qt\\test_output.txt'
        with open(filepath, 'r') as content_file:
            content = content_file.read().decode("hex")
            for match in re.finditer('[\02](.*?)[\03]', content, re.S):
                res = match.group(1)
                complete =  '\x02' + res + '\x03'
                # time.sleep(0.3) <-- if i uncomment this line, it work's!!!
                self.ser.write(complete)

if __name__ == "__main__": #wenn Modul direkt ausgefuehrt wird
    d = digiRealTest()
    d.main()

My Python script for receiving the data sent from the microcontroller:

import Queue
import threading
import serial

class mySerial(threading.Thread):
    def __init__(self, queue):
        super(mySerial, self).__init__()

        self.queue = queue #the received data is put in a queue

        self.buffer = ''

        #configure serial connection
        self.ser = serial.Serial(timeout = 0, port = 3, baudrate=9600)

    def run(self):              
        while True:
            self.buffer += self.ser.read(self.ser.inWaiting()) #read all char in buffer
            if '\n' in self.buffer: #split data line by line and store it in var
                var, self.buffer = self.buffer.split('\n')[-2:]
                self.queue.put(var) #put received line in the queue
        time.sleep(0.01)   #do not monopolize CPU



class Base():
    def __init__(self):
        self.queue = Queue.Queue(0) #create a new queue
        self.ser = mySerial(self.queue) 
        self.ser.start() #run thread


    def main(self   ):
        while(True):
            try:
                var = self.queue.get(False) #try to fetch a value from queue
            except Queue.Empty: 
                pass #if it is empty, do nothing
            else:
                print(var) 


if __name__ == '__main__':
    b = Base() 
    b.main()

I don't know why, but if I uncomment the line #time.sleep(0.3) in the sending script, everything works fine and I get the expected data back. So to me, it seems, that somehow my script for reading the data from the serial port is too slow....but why?

Answer 1

Your receiver's split is throwing lines away. This should work better:

def run(self):              
    while True:
        self.buffer += self.ser.read(self.ser.inWaiting()) #read all char in buffer
        while '\n' in self.buffer: #split data line by line and store it in var
            var, self.buffer = self.buffer.split('\n', 1)
            self.queue.put(var) #put received line in the queue
        time.sleep(0.01)   #do not monopolize CPU

You could also get rid of the polling/sleeping by getting rid of the timeout when you create the serial port and ask for 1 byte when there is nothing in the queue.

class mySerial(threading.Thread):
    def __init__(self, queue):
        super(mySerial, self).__init__()
        self.queue = queue #the received data is put in a queue
        self.buffer = ''
        #configure serial connection
        self.ser = serial.Serial(port = 3, baudrate=9600)

    def run(self):              
        while True:
            self.buffer += self.ser.read(self.ser.inWaiting() or 1) #read all char in buffer
            while '\n' in self.buffer: #split data line by line and store it in var
                var, self.buffer = self.buffer.split('\n', 1)
                self.queue.put(var) #put received line in the queue

Answer 2

when you sleep nothing at all happens ... a serial buffer is fairly small and probably just fills up in the time it takes to sleep(on the reciever) ... once its full it cant take any more until you read it so the other bits are lost(or maybe they overwrite the bits in the buffer) ... since serial does not have some handshake to say hey I recieved these bits

when you uncomment the sleep line from the sender that lets it sleep until the receiver has read the whole stream and cleared it out for more data

or maybe try

var, self.buffer = self.buffer.split('\n',1)  #this will only split once

because information can be lost with

var, self.buffer = self.buffer.split('\n')[-2:]

if you have recieved 2(or more) \n