'argparse' with optional positional arguments that start with dash

Question

We're trying to build a wrapper script over a command line tool we're using. We would like to set some tool arguments based on options in our wrapper scripts. We would also like to have the possibility to pass native arguments to the command line tool directly as they are written on the command line.

Here is what we came up with:

import argparse

parser = argparse.ArgumentParser()

parser.add_argument('positional')
parser.add_argument('-f', '--foo', action='store_true')
parser.add_argument('-b', '--bar', action='store_true')

parser.add_argument('native_arg', nargs='*')

args = parser.parse_args()
print (args)

positional is mandatory. Based on the options -f and -b we would add some extra options to our tool call. Anything that is left afterwards (if anything) should be treated as a native tool argument and given to the tool directly. Calling our script with -h produces the following usage:

usage: test.py [-h] [-f] [-b] positional [native_arg [native_arg ...]]

The trick is that these native arguments are themselves options for the tool and contain leading dashes, for example -native0 and -native1. We already know about the trick with the double dash to stop argparse from looking for more options. The following call:

./test.py pos -- -native0 -native1

produces the expected parsed arguments:

Namespace(bar=False, foo=False, native_arg=['-native0', '-native1'], positional='pos')

Trying to add an option after the first positional argument doesn't work, though. More specifically, the following call:

./test.py pos --foo -- -native0 -native1

produces the following output:

usage: [...shortened...]
test.py: error: unrecognized arguments: -- -native0 -native1

Putting the optional arguments before the positionals:

./test.py --foo pos -- -native0 -native1

seems to work, as the following is printed:

Namespace(bar=False, foo=True, native_arg=['-native0', '-native1'], positional='pos')

Even stranger, changing the value of nargs for native_arg to '+' works in all the above situations (with the caveat, of course, that at least one native_arg is expected).

Are we doing something wrong in our Python code or is this some kind of argparse bug?

Answer 1

argparse does have a hard time when you mix non-required positional arguments with optional arguments (see https://stackoverflow.com/a/47208725/1399279 for details into the bug report). Rather than suggesting a way to solve this issue, I am going to present an alternative approach.

You should check out the parse_known_args method, which was created for the situation you describe (i.e. passing options to a wrapped tool).

In [1]: import argparse

In [2]: parser = argparse.ArgumentParser()

In [3]: parser.add_argument('positional')

In [4]: parser.add_argument('-f', '--foo', action='store_true')

In [5]: parser.add_argument('-b', '--bar', action='store_true')

In [6]: parser.parse_known_args(['pos', '--foo', '-native0', '-native1'])
Out[6]: (Namespace(bar=False, foo=True, positional='pos'), ['-native0', '-native1'])

Unlike parse_args, the output of parse_known_args is a two-element tuple. The first element is the Namespace instance you would expect to get from parse_args, and it contains all the attributes defined by calls to add_argument. The second element is a list of all the arguments not known to the parser.

I personally prefer this method because the user does not need to remember any tricks about how to call your program, or which option order does not result in errors.

Answer 2

This is a known issue (https://bugs.python.org/issue15112, argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional)

The parsing alternates handling positionals and optionals. When dealing with positionals it tries to handle as many as the input strings require. But an ? or * positional is satisfied with [], an empty list of strings. + on the other hand requires at least one string

./test.py pos --foo -- -native0 -native1

The parser gives 'pos' to positional, and [] to native-arg. Then it gives '--foo' to its optional. There aren't anymore positionals left to hand the remaining strings, so it raises the error.

The allocation of input strings is done with a stylized form of regex string matching. Imagine matching a pattern that looks like AA?.

To correct this, parser would have to look ahead, and delay handling native-arg. We've suggested patches but they aren't in production.

@SethMMorton's suggestion of using parse_known_args is a good one.

Earlier parsers (e.g. Optparse) handle all the flagged arguments, but return the rest, the positionals, as a undifferentiated list. It's up to the user to split that list. argparse has added the ability to name and parse positionals, but the algorithm works best with fixed nargs, and gets flaky with too many variable nargs.