Capture print output from multiple threads in python

Question

I have a function which inside lauches multiple threads running in parallel. Threads print something and I want to capture this output from external function. I tried the next code to capture the output:

import sys, io
stdout = sys.stdout
sys.stdout = io.StringIO()
threads_conn(connect, devices) #- here many threads starts with many print inside
output = sys.stdout.getvalue()
sys.stdout = stdout

print(output)

This code works. But the problem is that output is printed only when all theads complete. So I have a freeze. Once all threads are done - whole output is printed. What I would like to have - have output once it is generated inside - in other words in real-time. Now I have the whole outout buffer printed at once. How get all of the threads to output in real-time?

Answer 1

What you want to do is write your own TextIOBase class (or, if you want binary data, write your own RawIOBase and then wrap a stock TextIOWrapper around it), where you can put whatever behavior you want.

As you can see from the docs. all you need to implement for TextIOBase is detach, read, readline, and write. And the first three aren't relevant for what you're doing.

---

So, what should your write look like? Well, it depends on what you want to do.

it sounds like your goal is to tee everything to both real stdout and to a StringIO. If so, this is pretty trivial.

The only question is what you want to do if one of the targets raises an exception, or writes fewer bytes than the other, etc. Since an IOString is never going to do either of those, we can write something really dumb that just assumes that whatever real stdout did was the right thing to do.

class TeeTextIO(io.TextIOBase):
    def __init__(self, target):
        self.target = target
        self.stringio = io.StringIO()
    def write(self, s):
        writecount = self.target.write(s)
        self.stringio.write(s[:writecount])
        return writecount

And now:

stdout = sys.stdout
sys.stdout = TeeTextIO(sys.stdout)
threads_conn(connect, devices) #- here many threads starts with many print inside
output = sys.stdout.stringio.getvalue()
sys.stdout = stdout

Now, the output has gone to the real stdout as it came in, but it's also been stored in the StringIO for whatever you want to do with it later.

(Notice that this class will work with any TextIOBase, like a file you open, not just with stdout. It didn't cost us anything to make it general, so why not?)

---

What if you wanted to do something totally different, like spread each write randomly among 10 different files? It should be obvious:

class SpreadTextWriter(io.TextIOBase):
    def __init__(self, *files):
        self.files = files
    def write(self, s):
        return random.choice(self.files).write(s)