Pyserial buffer fills faster than I can read

Question

I am reading data from a microcontroller via serial, at a baudrate of 921600. I'm reading a large amount of ASCII csv data, and since it comes in so fast, the buffer get's filled and all the rest of the data gets lost before I can read it. I know I could manually edit the pyserial source code for serialwin32 to increase the buffer size, but I was wondering if there is another way around it?

I can only estimate the amount of data I will receive, but it is somewhere around 200kB of data.

Answer 1

Have you considered reading from the serial interface in a separate thread that is running prior to sending the command to uC to send the data?

This would remove some of the delay after the write command and starting the read. There are other SO users who have had success with this method, granted they weren't having buffer overruns.

If this isn't clear let me know and I can throw something together to show this.

EDIT

Thinking about it a bit more, if you're trying to read from the buffer and write it out to the file system even the standalone thread might not save you. To minimize the processing time you might consider reading say 100 bytes at a time serial.Read(size=100) and pushing that data into a Queue to process it all after the transfer has completed

Pseudo Code Example

def thread_main_loop(myserialobj, data_queue):
    data_queue.put_no_wait(myserialobj.Read(size=100))

def process_queue_when_done(data_queue):
    while(1):
        if len(data_queue) > 0:
            poped_data = data_queue.get_no_wait()
            # Process the data as needed
        else:
            break;

Answer 2

There's a "Receive Buffer" slider that's accessible from the com port's Properties Page in Device Manager. It is found by following the Advanced button on the "Port Settings" tab.

More info:

http://support.microsoft.com/kb/131016 under heading Receive Buffer

http://tldp.org/HOWTO/Serial-HOWTO-4.html under heading Interrupts

Try knocking it down a notch or two.