Is there any way to instruct argparse (Python 2.7) to remove found arguments from sys.argv?

Question

I'm fully midstream in the development process for what is turning into a fairly substantial Python 2.7 project. Right now I have all of my unittest classes lumped together in their own module, tests.py, which stands at about 3300 lines. This is crazy big, impossible to navigate, all around bad practice, etc.

So, my current task is to refactor it into submodules. As part of this refactoring, I want to make it easy to just run a subset of the tests from the commandline. For example:

$ python tests.py --all                         <--- runs *all* tests
$ python tests.py --utils_base                  <--- runs all tests on .utils
$ python tests.py --utils_vector                <--- runs all tests on .utils.vector
$ python tests.py --utils_base --utils_vector   <--- runs all tests on .utils.vector & .utils

So, I started getting things set up with argparse. I got a basic ArgumentParser set up without a problem, and the help message showed just fine:

$ python tests.py -h
usage: tests.py [-h] [--all] [--utils_base]

optional arguments:
  -h, --help    show this help message and exit

Global Options:
  --all         Run all tests (overrides any other selections)

opan.base.utils Tests:
  --utils_base  Run all .utils tests

However, when I went to run some tests, it crashed out with an 'argument not recognized' error:

$ python tests.py --all
option --all not recognized
Usage: tests.py [options] [test] [...]

Options:
  -h, --help       Show this message
  -v, --verbose    Verbose output
  -q, --quiet      Minimal output
  -f, --failfast   Stop on first failure
  -c, --catch      Catch control-C and display results
  -b, --buffer     Buffer stdout and stderr during test runs

Examples:
  tests.py                               - run default set of tests
  tests.py MyTestSuite                   - run suite 'MyTestSuite'
  tests.py MyTestCase.testSomething      - run MyTestCase.testSomething
  tests.py MyTestCase                    - run all 'test*' test methods
                                               in MyTestCase

After some debugging, I finally realized that my commandline arguments to be parsed within tests.py were being retained in sys.argv and passed through to unittest.main. (In retrospect, a careful read of the Examples: section of the error message should have clued me in a lot sooner.)

So, to resolve the problem, I added the marked code in the following to scrub my custom arguments from sys.argv before passing control into unittest.main:

# Arguments for selecting test suites
ALL = 'all'
UTILS_BASE = 'utils_base'

# Collecting the args together for iteration later
test_args = [ALL, UTILS_BASE]

...

# Strip from sys.argv any test arguments that are present
for a in test_args:                 <---  
    str_arg = '--{0}'.format(a)     <--- ADDING THESE MAKES UNITTEST HAPPY
    if str_arg in sys.argv:         <---
        sys.argv.remove(str_arg)    <---

Is there any way to tell argparse to .remove arguments it finds from sys.argv, perhaps in the construction of the ArgumentParser? I've scoured the argparse doc page, but for the life of me can't find an option that'll do it.

Answer 1

You want parse_known_args()

from __future__ import print_function
import argparse
import sys

def main():
    print(sys.argv)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()

    parser.add_argument('--all', action='store_true')
    parser.add_argument('--utils_base', action='store_true')

    args, left = parser.parse_known_args()

    sys.argv = sys.argv[:1]+left

    main()

Although, I must ask. Why are you writing your own test runner? The unittest module allows you to run specific sets of tests from the cli:

# run test from a spefiic class
$ python -m unittest module.tests.group.TestSpecific
# all possible options
$ python -m unittest --help
Usage: python -m unittest [options] [tests]

Options:
  -h, --help       Show this message
  -v, --verbose    Verbose output
  -q, --quiet      Minimal output
  -f, --failfast   Stop on first failure
  -c, --catch      Catch control-C and display results
  -b, --buffer     Buffer stdout and stderr during test runs

Examples:
  python -m unittest test_module               - run tests from test_module
  python -m unittest module.TestClass          - run tests from module.TestClass
  python -m unittest module.Class.test_method  - run specified test method

[tests] can be a list of any number of test modules, classes and test
methods.

Alternative Usage: python -m unittest discover [options]

Options:
  -v, --verbose    Verbose output
  -f, --failfast   Stop on first failure
  -c, --catch      Catch control-C and display results
  -b, --buffer     Buffer stdout and stderr during test runs
  -s directory     Directory to start discovery ('.' default)
  -p pattern       Pattern to match test files ('test*.py' default)
  -t directory     Top level directory of project (default to
                   start directory)

For test discovery all test modules must be importable from the top
level directory of the project.

If you need even more flexibility in grouping and running tests, I'd suggest looking at nosetest