Subclassing Future Class

Question

I would like to subclass the Future class of the concurrent Python module.

The docs:

The Future class encapsulates the asynchronous execution of a callable. Future instances are created by Executor.submit().

The docs of Executor don't explain where it takes the Future class from.

... How can make Executor.submit() force to take my custom Future class?

Why do I need it?

I like OOP since it creates readable code. I would like the result to look like this:

for my_future in concurrent.futures.as_completed(...):
    my_future.my_custom_method()

Answer 1

Looking at the code, ProcessPoolExecutor.submit() and ThreadPollExecutor.submit(), Excutor.submit() returns an instance of Future, which is defined in conccurent.futures._base.Future.

So, here comes the trick. You can subclass and replace the original Future, and then add custom methods in the subclass.

It's doable, but not recommended. It's better to use composition rather than inheritance for this purpose. There is a good chapter on inheritance and composition in Learn Python the Hard Way

Back to the the question, here is a composition example:

class Myclass(object):
    def __init__(self, workers=3):
         self.executor = concurrent.futures.ProcessPoolExcutor(workers)

    def run(self, job):
        '''Job to be run in a Executor'''

    def submit(self, jobs):
        self.futures = [executor.submit(self, foo, job) for job in jobs]

    def done(self, result):
        '''Dealing with the result'''

    def harvest(self):
        for my_future in concurrent.futures.as_completed(self.futures):
            self.done(my_future.result())

And then you can subclass MyClass and implement different done method.

Answer 2

The use of the Future concrete class is hard-wired into Executor.submit() (whether for processes or threads). Therefore, I do not think it is possible to do exactly what you are asking. However, you can return any result from the callable passed to Executor.submit(). Therefore, put your custom methods in a custom return class:

class my_result(object):
    def my_custom_method(self):
        pass

def x():
    return my_result()

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(x), executor.submit(x)]
    for my_future in concurrent.futures.as_completed(futures):
        my_future.result().my_custom_method()
               # ^^^^^^^^^

Edit Or, if you really want your inner loop to be clean, change the last two lines to:

    for my_result in (f.result() for f in concurrent.futures.as_completed(futures)):
                   # ^^^^^^^^^^^^^^^^^^^^
        my_result.my_custom_method()

The generator expression (f.result() ... (futures)) takes the iterator of futures from as_completed and gives you an iterator of the results of those futures. You can then loop through those results.