Using asyncio for Non-async Functions in Python?

Question

Suppose there is a library that makes various database queries:

import time

def queryFoo():
    time.sleep(4)
    return "foo"

def queryBar():
    time.sleep(4)
    return "bar"

I want to execute those 2 queries concurrently without having to add async to the method signature or adding a decorator. These functions should not depend on asyncio at all.

What is the best way to utilize those non-async functions within asyncio?

I am looking for something of the form:

#I need an 'asyncWrapper'

results = asyncio.gather(asyncWrapper(queryFoo()), asyncWrapper(queryBar()))

Thank you in advance for your consideration and response.

Answer 1

If some function is blocking and not async by nature, only proper way to run it inside asyncio event loop is to run it inside thread using run_in_executor:

# Our example blocking functions
import time


def queryFoo():
    time.sleep(3)
    return 'foo'


def queryBar():
    time.sleep(3)
    return 'bar'


# Run them using asyncio
import asyncio
from concurrent.futures import ThreadPoolExecutor


_executor = ThreadPoolExecutor(10)


async def in_thread(func):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_executor, func)


async def main():
    results = await asyncio.gather(
        in_thread(queryFoo), 
        in_thread(queryBar),
    )

    print(results)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

It does job.

If you however want to avoid using threads only way to do it - is to rewrite queryFoo/queryBar to be async by nature.

Answer 2

I presume you are after concurrency and hopefully do not insist on using asyncio module itself in which case this little example could be helpful:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

def queryFoo():
    time.sleep(2)
    return "FOO"

def queryBar():
    time.sleep(4)
    return "BAR"

with ThreadPoolExecutor(max_workers=2) as executor:
    foo = executor.submit(queryFoo)
    bar = executor.submit(queryBar)
    results = [foo.result(), bar.result()]

print(results)

It runs both queryFoo() and queryBar() in parallel and collects their results in a list in order in which they've been mentioned in an assignment to results.