Run tests concurrently

Question

I would like to run several tests concurrently using asyncio (/curio/trio) and pytest, but I couldn't find any information on that. Do I need to schedule them myself? And if I do, is there a way to have a nice output that separates (sub-)tests cases?

Here is a small toy example for which I'm trying this:

import pytest
import time
import asyncio

pytestmark = pytest.mark.asyncio
expected_duration = 1
accepted_error = 0.1

async def test_sleep():
    start = time.time()
    time.sleep(expected_duration)
    duration = time.time() - start
    assert abs(duration-expected_duration) < accepted_error

async def test_async_sleep():
    start = time.time()
    await asyncio.sleep(expected_duration)
    duration = time.time() - start
    assert abs(duration-expected_duration) < accepted_error

Answer 1

Unfortunately, the way pytest works internally, you can't really run multiple tests at the same time under the same call to trio.run/asyncio.run/curio.run. (This is also good in some ways – it prevents state from leaking between tests, and with trio at least it lets you configure trio differently for different tests, e.g. setting one test to use the autojump clock while another test doesn't.)

Definitely the simplest option is to use pytest-xdist to run tests in separate threads. You can still use async internally inside each test – all these async libraries support running different loops in different threads.

If you really need to use async concurrency, then I think you'll have to write a single pytest test function, and then inside that function do your own scheduling and concurrency. If you do it this way, then from pytest's perspective there's only one test, so it won't be easy to get nice per-test output. I guess you could try using pytest-subtests?

Answer 2

There's https://github.com/willemt/pytest-asyncio-cooperative now.

Today 2020-03-25, the limitations are quite steep — you have to ensure your tests don't share anything (well, technically, don't share mutable state) and you can't use mock.patch (technically don't mock anything another test might use).

You can follow the discussion at https://github.com/pytest-dev/pytest-asyncio/issues/69, I believe it is hard, but possible to come up with a way mark each fixture to allow or disallow concurrent use, and schedule tests to preserve these restrictions.