Using unittest to test argparse - exit errors

Question

Going off of Greg Haskin's answer in this question, I tried to make a unittest to check that argparse is giving the appropriate error when I pass it some args that are not present in the choices. However, unittest generates a false positive using the try/except statement below.

In addition, when I make a test using just a with assertRaises statement, argparse forces the system exit and the program does not execute any more tests.

I would like to be able to have a test for this, but maybe it's redundant given that argparse exits upon error?

#!/usr/bin/env python3

import argparse
import unittest

class sweep_test_case(unittest.TestCase):
    """Tests that the merParse class works correctly"""

    def setUp(self):
        self.parser=argparse.ArgumentParser()
        self.parser.add_argument(
            "-c", "--color",
            type=str,
            choices=["yellow", "blue"],
            required=True)

    def test_required_unknown_TE(self):
        """Try to perform sweep on something that isn't an option.
        Should return an attribute error if it fails.
        This test incorrectly shows that the test passed, even though that must
        not be true."""
        args = ["--color", "NADA"]
        try:
            self.assertRaises(argparse.ArgumentError, self.parser.parse_args(args))
        except SystemExit:
            print("should give a false positive pass")

    def test_required_unknown(self):
        """Try to perform sweep on something that isn't an option.
        Should return an attribute error if it fails.
        This test incorrectly shows that the test passed, even though that must
        not be true."""
        args = ["--color", "NADA"]
        with self.assertRaises(argparse.ArgumentError):
            self.parser.parse_args(args)

if __name__ == '__main__':
    unittest.main()

Errors:

Usage: temp.py [-h] -c {yellow,blue}
temp.py: error: argument -c/--color: invalid choice: 'NADA' (choose from 'yellow', 'blue')
E
usage: temp.py [-h] -c {yellow,blue}
temp.py: error: argument -c/--color: invalid choice: 'NADA' (choose from 'yellow', 'blue')
should give a false positive pass
.
======================================================================
ERROR: test_required_unknown (__main__.sweep_test_case)
Try to perform sweep on something that isn't an option.
----------------------------------------------------------------------
Traceback (most recent call last): #(I deleted some lines)
  File "/Users/darrin/anaconda/lib/python3.5/argparse.py", line 2310, in _check_value
    raise ArgumentError(action, msg % args)
argparse.ArgumentError: argument -c/--color: invalid choice: 'NADA' (choose from 'yellow', 'blue')

During handling of the above exception, another exception occurred:

Traceback (most recent call last): #(I deleted some lines)
  File "/anaconda/lib/python3.5/argparse.py", line 2372, in exit
    _sys.exit(status)
SystemExit: 2

Answer 1

The trick here is to catch SystemExit instead of ArgumentError. Here's your test rewritten to catch SystemExit:

#!/usr/bin/env python3

import argparse
import unittest

class SweepTestCase(unittest.TestCase):
    """Tests that the merParse class works correctly"""

    def setUp(self):
        self.parser=argparse.ArgumentParser()
        self.parser.add_argument(
            "-c", "--color",
            type=str,
            choices=["yellow", "blue"],
            required=True)

    def test_required_unknown(self):
        """ Try to perform sweep on something that isn't an option. """
        args = ["--color", "NADA"]
        with self.assertRaises(SystemExit):
            self.parser.parse_args(args)

if __name__ == '__main__':
    unittest.main()

That now runs correctly, and the test passes:

$ python scratch.py
usage: scratch.py [-h] -c {yellow,blue}
scratch.py: error: argument -c/--color: invalid choice: 'NADA' (choose from 'yellow', 'blue')
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

However, you can see that the usage message is getting printed, so your test output is a bit messed up. It might also be nice to check that the usage message contains "invalid choice".

You can do that by patching sys.stderr:

#!/usr/bin/env python3

import argparse
import unittest
from io import StringIO
from unittest.mock import patch


class SweepTestCase(unittest.TestCase):
    """Tests that the merParse class works correctly"""

    def setUp(self):
        self.parser=argparse.ArgumentParser()
        self.parser.add_argument(
            "-c", "--color",
            type=str,
            choices=["yellow", "blue"],
            required=True)

    @patch('sys.stderr', new_callable=StringIO)
    def test_required_unknown(self, mock_stderr):
        """ Try to perform sweep on something that isn't an option. """
        args = ["--color", "NADA"]
        with self.assertRaises(SystemExit):
            self.parser.parse_args(args)
        self.assertRegexpMatches(mock_stderr.getvalue(), r"invalid choice")


if __name__ == '__main__':
    unittest.main()

Now you only see the regular test report:

$ python scratch.py
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

For pytest users, here's the equivalent that doesn't check the message.

import argparse

import pytest


def test_required_unknown():
    """ Try to perform sweep on something that isn't an option. """
    parser=argparse.ArgumentParser()
    parser.add_argument(
        "-c", "--color",
        type=str,
        choices=["yellow", "blue"],
        required=True)
    args = ["--color", "NADA"]

    with pytest.raises(SystemExit):
        parser.parse_args(args)

Pytest captures stdout/stderr by default, so it doesn't pollute the test report.

$ pytest scratch.py
================================== test session starts ===================================
platform linux -- Python 3.6.7, pytest-3.5.0, py-1.7.0, pluggy-0.6.0
rootdir: /home/don/.PyCharm2018.3/config/scratches, inifile:
collected 1 item                                                                         

scratch.py .                                                                       [100%]

================================ 1 passed in 0.01 seconds ================================

You can also check the stdout/stderr contents with pytest:

import argparse

import pytest


def test_required_unknown(capsys):
    """ Try to perform sweep on something that isn't an option. """
    parser=argparse.ArgumentParser()
    parser.add_argument(
        "-c", "--color",
        type=str,
        choices=["yellow", "blue"],
        required=True)
    args = ["--color", "NADA"]

    with pytest.raises(SystemExit):
        parser.parse_args(args)

    stderr = capsys.readouterr().err
    assert 'invalid choice' in stderr

As usual, I find pytest easier to use, but you can make it work in either one.

Answer 2

I know this is an old question but just to expand on @don-kirkby's answer of looking for SystemExit – but without having to use pytest or patching – you can wrap the testcode in contextlib.redirect_stderr, if you want to assert something about the error message:

    import contextlib
    from io import StringIO
    import unittest
    class MyTest(unittest.TestCase):
        def test_foo(self):
            ioerr = StringIO()
            with contextlib.redirect_stderr(ioerr):
                with self.assertRaises(SystemExit) as err:
                    foo('bad')
            self.assertEqual(err.exception.code, 2)
            self.assertIn("That is a 'bad' thing", ioerr.getvalue())