Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pytest - error vs fail

Im migrating from PyUnit to Pytest, and I found, that Pytest, unlike PyUnit, does not distinguish fails and errors in test report in quick report while running tests (where dots are printed). How to teach Pytest do do it?

UPDATE

Seems like it is valid only for PyUnit tests executed with Pytest, thanks to flub for the clue.

Code:

import unittest

class TestErrorFail(unittest.TestCase):
    def test_error(self):
        raise Exception('oops')

    def test_fail(self):
        self.assertTrue(False)

Output:

================================ test session starts =================================
platform linux2 -- Python 2.7.3 -- py-1.4.20 -- pytest-2.5.2
plugins: django
collected 2 items 

sometests.py FF

====================================== FAILURES ======================================
______________________________ TestErrorFail.test_error ______________________________

self = <sometests.TestErrorFail testMethod=test_error>

    def test_error(self):
>       raise Exception('oops')
E       Exception: oops

sometests.py:5: Exception
______________________________ TestErrorFail.test_fail _______________________________

self = <sometests.TestErrorFail testMethod=test_fail>

    def test_fail(self):
>       self.assertTrue(False)
E       AssertionError: False is not true

sometests.py:8: AssertionError
============================== 2 failed in 0.69 seconds ==============================
like image 998
Gill Bates Avatar asked Feb 02 '14 17:02

Gill Bates


People also ask

How do I fix pytest error?

Hey, The error who are facing can be result of different version of pytest. Please try using older version of pytest by uninstalling it from cmd using the command "pip uninstall pytest" and installing the older version again and try running the code.

What does pytest Mark skip do?

Pytest markers The markers are attached to a test function in the form of function decorators. There are some inbuilt markers like xfail , skip and so on. The skip is one such marker provided by pytest that is used to skip test functions from executing.

How do I debug pytest?

Debug Mode in pytest can be triggered in a few different ways: Your test raises an exception after passing the "--pdb" option to "pytest" . The moment your test begins after passing the "--trace" option to "pytest" . Calling "pdb.


2 Answers

For pytest, any uncaught exception thrown in a test function is a failure, including but not limited to assertion errors.

Error is reserved for a failure in a fixture.
Uncaught exceptions in a named pytest fixture, as in flub's example, or in xUnit style setup/teardown fixtures, result in an Error instead of a failure.

I personally like the distinction.
A Failure indicates that the test failed in some way.
An Error indicates that you couldn't get to the point of doing a proper test.

Note that an Error will happen even in the case where the exception is in the teardown.
In this case, you completed the test, and the teardown failed in some way.

like image 95
Okken Avatar answered Sep 20 '22 15:09

Okken


To the best of my knowledge py.test does distinguish failures and errors, consider this example:

import pytest

def test_fail():
    assert 1 == 2

@pytest.fixture
def fix():
    raise Exception('oops')

def test_error(fix):
    assert fix == 2

Running this test module gives one failure and one error:

================ test session starts =========================
platform linux2 -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2
plugins: timeout, capturelog, xdist
collected 2 items 

../../tmp/test_foo.py FE

======================= ERRORS ===============================
_________________ ERROR at setup of test_error _______________

    @pytest.fixture
    def fix():
>       raise Exception('oops')
E       Exception: oops

/tmp/test_foo.py:8: Exception
====================== FAILURES ==============================
__________________________ test_fail ____________________________

    def test_fail():
>       assert 1 == 2
E       assert 1 == 2

/tmp/test_foo.py:4: AssertionError
============= 1 failed, 1 error in 0.12 seconds ================

UPDATE

Note however that py.test considers any exception raised during the exception itself a normal failure. This is actually a good thing, generally you want to be able to fail your test with an exception which is not an AssertionError (or subclass thereof). In the example above you will find that the error condition is triggered by raising the exception in the fixture rather then during the test.

However trying this out with a UnitTest class it turns out that raising an exception in the .setUp() method is does result in a failure rather then error. This is probably a bug and you could report it as such if you like.

like image 26
flub Avatar answered Sep 19 '22 15:09

flub