How to select a subset of tests in pytest using custom markers on params

Question

I wonder how to select a subset of my tests using pytest custom markers

A simple test works as expected:

Code with one marked parameter

import pytest

@pytest.mark.parametrize('a', [pytest.mark.my_custom_marker(0), 1])
@pytest.mark.parametrize('b', [0, 1])
def test_increment(a, b):
    pass

If I only want to run test marked with 'my_custom_marker'

Output

$ pytest test_machin.py -m my_custom_marker --collect-only
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /home/mvelay/workspace/sandbox, inifile: 
plugins: hypothesis-3.6.0, html-1.12.0, xdist-1.15.0, timeout-1.0.0
collected 4 items 
<Module 'test_machin.py'>
  <Function 'test_increment[0-0]'>
  <Function 'test_increment[0-1]'>

But as soon as I try to test multi marked parameters, I am facing an issue

Code with two marked paremeters

import pytest

@pytest.mark.parametrize('a', [pytest.mark.my_custom_marker(0), 1])
@pytest.mark.parametrize('b', [pytest.mark.my_custom_marker(0), 1])
def test_increment(a, b):
    pass

Output

$ pytest -m my_custom_marker test_machin.py --collect-only
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /home/mvelay/workspace/sandbox, inifile: 
plugins: hypothesis-3.6.0, html-1.12.0, xdist-1.15.0, timeout-1.0.0
collected 4 items 
<Module 'test_machin.py'>
  <Function 'test_increment[0-0]'>
  <Function 'test_increment[0-1]'>
  <Function 'test_increment[1-0]'>

I was expected that only [0-0] combination ran.

Is there a way to do it in an elegant way ?

Answer 1

You can use two different markers as follows:

import pytest

@pytest.mark.parametrize('a', [pytest.mark.marker_a(0), 1])
@pytest.mark.parametrize('b', [pytest.mark.marker_b(0), 1])
def test_increment(a, b):
    pass

And specify a mark expression:

$ pytest test_machin.py -m "marker_a and marker_b" --collect-only
============= test session starts ===============================
platform darwin -- Python 2.7.10, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /Users/yangchao/Code, inifile:
collected 4 items
<Module 'test_machin.py'>
  <Function 'test_increment[0-0]'>
============= 3 tests deselected =================================
============= 3 deselected in 0.00 seconds =======================

Answer 2

The answer here is you are abusing Marks. Test Runners are generally built to test the entire permutation space. You will find that everything in pytest is geared towards building the cartesian product. This is because people generally want to test as much as possible while writing the least amount of code. When considering this you will find yourself to be the anomaly and this to be an XY problem

This doesn't mean there isn't an answer. So what do you really want to do here? Here is what you are asking ...

I wonder how to select a subset of my tests using pytest custom markers

But I wonder if what you are saying is...

I want to execute my test with different sets of parameters, on-demand using marks.

This is a much easier thing to do. Here it is in its simplest form ...

import pytest

@pytest.mark.A
def test_add():

  a = 2
  b = 1
  result = add(a,b)
  assert result == 3

@pytest.mark.B
def test_add_with_failure():

  a = 1
  b = 2    
  add(a,b)  
  #expect it to fail

def add(a, b):
  assert a>b 
  return a+b

Now each of these test sets can be invoked from the command line with marks.

py.test -m A tests.py py.test -m B tests.py

This is what the mark's original job was. To locate a specific test or group of tests and execute it(them). If you are trying to do something beyond finding a specific test to execute then you are fighting the framework.

If you want something a little more canonical pytest you can take some of the boilerplate and fixture it on up. This positions you to add lots of permutations. Like so...

@pytest.fixture(params=[(2,1,3)])
def always_pass(request):
    return request.param

@pytest.mark.A
def test_add(always_pass):

  a, b, expected = always_pass
  result = add(a,b)
  assert result == expected

Or you can go full on and build the all the tests into one big, composable fixture...

@pytest.fixture(params=[
    (2, 1, 3),
    (1, 2, 'fail')
    ])
def all_the_things(request):
    return request.param

@pytest.mark.A
def test_add(all_the_things):

  a, b, expected = all_the_things

  if expected == 'fail'
    with pytest.raises(AssertionError):
        result = add(a,b)
  else:
    assert result == expected

The point here is that the framework is telling you something when it stacks everything together that way. Its saying "there is a better way". Don't fight the framework, embrace it! If you look closer at parameterize while considering the principles described above you'll see its really a feature to allow you more fine, grained composability. Such that you can build a cartesian product of individual parameters using decorator stacking and marks.