Python, non-blocking threads

Question

There are a lot of tutorials etc. on Python and asynchronous coding techniques, but I am having difficulty filtering the through results to find what I need. I am new to Python, so that doesn't help.

Setup

I currently have two objects that look sort of like this (please excuse my python formatting):

class Alphabet(parent):
    def init(self, item):
        self.item = item

    def style_alphabet(callback):
        # this method presumably takes a very long time, and fills out some properties
        # of the Alphabet object
        callback()


class myobj(another_parent):
    def init(self):
        self.alphabets = []
        refresh()
        
    def foo(self):
        for item in ['a', 'b', 'c']:
            letters = new Alphabet(item)
            self.alphabets.append(letters)
        self.screen_refresh()

        for item in self.alphabets
            # this is the code that I want to run asynchronously. Typically, my efforts
            # all involve passing item.style_alphabet to the async object / method
            # and either calling start() here or in Alphabet
            item.style_alphabet(self.screen_refresh)
            
    def refresh(self):
        foo()
        # redraw screen, using the refreshed alphabets
        redraw_screen()

    def screen_refresh(self):
        # a lighter version of refresh()
        redraw_screen()

The idea is that the main thread initially draws the screen with incomplete Alphabet objects, fills out the Alphabet objects, updating the screen as they complete.

I've tried a number of implementations of threading.Tread, Queue.Queue, and even futures, and for some reason they either haven't worked, or they have blocked the main thread. so that the initial draw doesn't take place.

A few of the async methods I've attempted:

class Async (threading.Thread):
    def __init__(self, f, cb):
        threading.Thread.__init__(self)
        self.f  = f
        self.cb = cb

    def run(self):
        self.f()
        self.cb()

---

def run_as_thread(f):
    # When I tried this method, I assigned the callback to a property of "Alphabet"
    thr = threading.Thread(target=f)
    thr.start()

---

def run_async(f, cb):
    pool = Pool(processes=1)
    result = pool.apply_async(func=f, args=args, callback=cb)

Answer 1

I ended up writing a thread pool to deal with this use pattern. Try creating a queue and handing a reference off to all the worker threads. Add task objects to the queue from the main thread. Worker threads pull objects from the queue and invoke the functions. Add an event to each task to be signaled on the worker thread at task completion. Keep a list of task objects on the main thread and use polling to see if the UI needs an update. One can get fancy and add a pointer to a callback function on the task objects if needed.

My solution was inspired by what I found on Google: http://code.activestate.com/recipes/577187-python-thread-pool/

I kept improving on that design to add features and give the threading, multiprocessing, and parallel python modules a consistent interface. My implementation is at:

https://github.com/nornir/nornir-pools

Docs:

http://nornir.github.io/packages/nornir_pools.html

If you are new to Python and not familiar with the GIL I suggest doing a search for Python threading and the global interpreter lock (GIL). It isn’t a happy story. Generally I find I need to use the multiprocessing module to get decent performance.

Hope some of this helps.