Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

running a test suite (an arbitrary collection of tests) with py.test

I'm using py.test to build functional test framework, so I need to be able to specify the exact tests to be run. I understand the beauty of dynamic test collection, but I want to be able to run my test environment health checks first, then run my regression tests after; that categorization does not preclude tests in these sets being used for other purposes.

The test suites will be tied to Jenkins build projects. I'm using osx, python 2.7.3, py.test 2.3.4.

So I have a test case like the following:

# sample_unittest.py
import unittest, pytest

class TestClass(unittest.TestCase):

    def setUp(self):
        self.testdata = ['apple', 'pear', 'berry']

    def test_first(self):
        assert 'apple' in self.testdata

    def test_second(self):
        assert 'pear' in self.testdata

    def tearDown(self):
        self.testdata = []

def suite():
    suite = unittest.TestSuite()
    suite.addTest(TestClass('test_first'))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())

And I have a test suite like this:

# suite_regression.py
import unittest, pytest
import functionaltests.sample_unittest as sample_unittest

# set up the imported tests
suite_sample_unittest = sample_unittest.suite()

# create this test suite
suite = unittest.TestSuite()
suite.addTest(suite_sample_unittest)

# run the suite
unittest.TextTestRunner(verbosity=2).run(suite)

If I run the following from the command line against the suite, test_first runs (but I don't get the additional information that py.test would provide):

python functionaltests/suite_regression.py -v

If I run the following against the suite, 0 tests are collected:

py.test functionaltests/suite_regression.py

If I run the following against the testcase, test_first and test_second run:

py.test functionaltests/sample_unittest.py -v

I don't see how doing py.test with keywords will help organize tests into suites. Placing testcases into a folder structure and running py.test with folder options won't let me organize tests by functional area.

So my questions:

  1. Is there a py.test mechanism for specifying arbitrary groupings of tests in a re-usable format?
  2. Is there a way to use unittest.TestSuite from py.test?

EDIT: So I tried out py.test markers, which lets me flag test functions and test methods with an arbitrary label, and then filter for that label at run time.

# conftest.py
import pytest

# set up custom markers
regression = pytest.mark.NAME
health = pytest.mark.NAME

And my updated test case:

# sample_unittest.py
import unittest, pytest

class TestClass(unittest.TestCase):

    def setUp(self):
        self.testdata = ['apple', 'pear', 'berry']

    @pytest.mark.healthcheck
    @pytest.mark.regression
    def test_first(self):
        assert 'apple' in self.testdata

    @pytest.mark.regression
    def test_second(self):
        assert 'pear' in self.testdata

    def tearDown(self):
        self.testdata = []

def suite():
    suite = unittest.TestSuite()
    suite.addTest(TestClass('test_first'))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite()) 

So running the following command collects and runs test_first:

py.test functionaltests/sample_unittest.py -v -m healthcheck

And this collects and runs test_first and test_second:

py.test functionaltests/sample_unittest.py -v -m regression

So back to my questions: markers is a partial solution, but I still don't have a way to control the execution of collected marked tests.

like image 567
sophe Avatar asked Mar 28 '13 20:03

sophe


People also ask

How do I run test suite in pytest?

Pytest provides two ways to run the subset of the test suite. Select tests to run based on substring matching of test names. Select tests groups to run based on the markers applied.

What is a test suite in Python?

A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together. test runner. A test runner is a component which orchestrates the execution of tests and provides the outcome to the user.

How do you run multiple test cases in Python?

If they have not provided their environment to run test cases in a loop. And locally if You are running code for python compiler It would be useful. you can simply use a while loop or range function of python.


2 Answers

No need to use markers in this case: setting the @pytest.mark.incremental on your py.test test class will force the execution order to the declaration order:

# sequential.py
import pytest

@pytest.mark.incremental
class TestSequential:

    def test_first(self):
        print('first')

    def test_second(self):
        print('second')

    def test_third(self):
        print('third')

Now running it with

pytest -s -v sequential.py

produces the following output:

=========== test session starts ===========
collected 3 items

sequential.py::TestSequential::test_first first
PASSED
sequential.py::TestSequential::test_second second
PASSED
sequential.py::TestSequential::test_third third
PASSED
=========== 3 passed in 0.01 seconds ===========
like image 191
ccpizza Avatar answered Oct 18 '22 17:10

ccpizza


I guess it's a bit late now but I just finished up an interactive selection plugin with docs here:

https://github.com/tgoodlet/pytest-interactive

I actually use the hook Holger mentioned above.

It allows you to choose a selection of tests just after the collection phase using IPython. Ordering the tests is pretty easy using slices, subscripts, or tab-completion if that's what you're after. Note that it's an interactive tool meant for use during development and not so much for automated regression runs.

For persistent ordering using marks I've used pytest-ordering which is actually quite useful especially if you have baseline prerequisite tests in a long regression suite.

like image 43
goodboy Avatar answered Oct 18 '22 16:10

goodboy