Is it possible to execute the same test in parallel in pytest?

Question

I understand that you can parameterize a test to repeat the test with different set of parameters. Also I know that different tests in a test file can be run in parallel using the -n, but I would like to execute the same set of tests in parallel. Is it possible in pytest?

For example:

import pytest

@pytest.fixture()
def user_number(worker_id):
    return "user number : %s" %worker_id

def test_add(user_number):
    print("Adding 1+1 and returning the result and user number: {}".format(user_number))
    return 1+1

def test_subtract():
    print("subtracting 2-1 and returning the result and user number: {}".format(user_number))
    return 2-1

If I run the following command: py.test -n 3 -s -v parallel_users.py

In the result, test_add() and test_subtract() are run in parallel as shown below:

[gw1] PASSED parallel_users.py::test_subtract 
[gw0] PASSED parallel_users.py::test_add

How can I get test_add() and test_subtract() run twice, something like below:

[gw1] PASSED parallel_users.py::test_add, test_subtract 
[gw0] PASSED parallel_users.py::test_add, test_subtract 

Answer 1

Update:

I guess the closest to what the OP is looking for is using the each distscope. Using it will execute the test selection n times, once per each process:

$ pytest -n3 -v --dist=each
...
gw0 [2] / gw1 [2] / gw2 [2]
scheduling tests via EachScheduling

test_main.py::test_add 
[gw1] [ 50%] PASSED test_main.py::test_add 
[gw0] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [100%] PASSED test_main.py::test_subtract 
[gw0] [100%] PASSED test_main.py::test_subtract 
[gw1] [100%] PASSED test_main.py::test_subtract 

Both test_add and test_subtract were executed once in each worker gw0, gw1 and gw2, summing up to three executions for each test.

Old answer

To repeat test execution, add a hook in your conftest.py:

def pytest_collection_modifyitems(items):
    numrepeats = 2
    items.extend(items * (numrepeats - 1))

This will copy each of the tests collected for the execution numrepeats times. Example run:

$ pytest test_spam.py -v -n3
============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .pytest_cache
rootdir: /private/tmp, inifile:
plugins: xdist-1.22.2, forked-0.2, dependency-0.3.2, cov-2.5.1
[gw0] darwin Python 3.6.4 cwd: /private/tmp
[gw1] darwin Python 3.6.4 cwd: /private/tmp
[gw2] darwin Python 3.6.4 cwd: /private/tmp
[gw0] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [4] / gw1 [4] / gw2 [4]
scheduling tests via LoadScheduling

test_spam.py::test_add
test_spam.py::test_subtract
test_spam.py::test_add
[gw0] [ 25%] PASSED test_spam.py::test_add
[gw1] [ 50%] PASSED test_spam.py::test_subtract
[gw2] [ 50%] PASSED test_spam.py::test_add
test_spam.py::test_subtract
[gw0] [ 50%] PASSED test_spam.py::test_subtract

=========================== 4 passed in 0.63 seconds ===========================

If you want to make it configurable, add a custom cli argument:

import pytest

def pytest_addoption(parser):
    parser.addoption('--numrepeats', action='store', type=int, default=1)

def pytest_collection_modifyitems(items):
    numrepeats = pytest.config.getoption('--numrepeats')
    items.extend(items * (numrepeats - 1))

Now you can call your tests with --numrepeats, for example pytest --numrepeats 5.

---

As for batching tests per process (second part of your question), pytest-xdist doesn't support it yet, see this issue and all the stuff linked to it. Recently, some rudimentary support like executing tests in a single module or class in a separate process was added:

--dist=distmode       set mode for distributing tests to exec environments.
                      each: send each test to all available environments.
                      load: load balance by sending any pending test to any
                      available environment. loadscope: load balance by
                      sending pending groups of tests in the same scope to
                      any available environment. loadfile: load balance by
                      sending test grouped by file to any available
                      environment. (default) no: run tests inprocess, don't
                      distribute.

However, if you want to loadbalance the tests based on some custom condition, there's no way other than writing your own scheduler impl.