How to control the incremental test case in Pytest

Question

@pytest.mark.incremental
class Test_aws():

    def test_case1(self):
        ----- some code here ----
        result = someMethodTogetResult
        assert result[0] == True
        orderID = result[1]

    def test_case2(self):
        result = someMethodTogetResult # can be only perform once test case 1 run successfully.
        assert result == True

    def test_deleteOrder_R53HostZonePrivate(self):
        result = someMethodTogetResult
        assert result[0] == True

The current behavior is if test 1 passes then test 2 runs and if test 2 passes then test 3 runs.

What I need is: If test_case 3 should be run if test_case 1 passed. test_case 2 should not change any behavior. Any thoughts here?

Answer 1

I guess you are looking for pytest-dependency which allows setting conditional run dependencies between tests. Example:

import random
import pytest


class TestAWS:

    @pytest.mark.dependency
    def test_instance_start(self):
        assert random.choice((True, False))

    @pytest.mark.dependency(depends=['TestAWS::test_instance_start'])
    def test_instance_stop(self):
        assert random.choice((True, False))

    @pytest.mark.dependency(depends=['TestAWS::test_instance_start'])
    def test_instance_delete(self):
        assert random.choice((True, False))

test_instance_stop and test_instance_delete will run only if test_instance_start succeeds and skip otherwise. However, since test_instance_delete does not depend on test_instance_stop, the former will execute no matter what the result of the latter test is. Run the example test class several times to verify the desired behaviour.

Answer 2

To complement hoefling's answer, another option is to use pytest-steps to perform incremental testing. This can help you in particular if you wish to share some kind of incremental state/intermediate results between the steps.

However it does not implement advanced dependency mechanisms like pytest-dependency, so use the package that better suits your goal.

With pytest-steps, hoefling's example would write:

import random
from pytest_steps import test_steps, depends_on

def step_instance_start():
    assert random.choice((True, False))

@depends_on(step_instance_start)
def step_instance_stop():
    assert random.choice((True, False))

@depends_on(step_instance_start)
def step_instance_delete():
    assert random.choice((True, False))

@test_steps(step_instance_start, step_instance_stop, step_instance_delete)
def test_suite(test_step):
    # Execute the step
    test_step()

EDIT: there is a new 'generator' mode to make it even easier:

import random
from pytest_steps import test_steps, optional_step

@test_steps('step_instance_start', 'step_instance_stop', 'step_instance_delete')
def test_suite():
    # First step (Start)
    assert random.choice((True, False))
    yield

    # Second step (Stop)
    with optional_step('step_instance_stop') as stop_step:
        assert random.choice((True, False))
    yield stop_step

    # Third step (Delete)
    with optional_step('step_instance_delete') as delete_step:
        assert random.choice((True, False))
    yield delete_step

Check the documentation for details. (I'm the author of this package by the way ;) )