Allow positional command-line arguments with nargs to be seperated by a flag

Question

I have a program using argparse. It takes 1 required positional argument, 1 optional positional argument, and 1 flag argument.

Something like:

usage: test.py [-h] [-a A] b [c]

So, I tried using this:

parser = argparse.ArgumentParser()

parser.add_argument('-a')
parser.add_argument('b')
parser.add_argument('c', nargs='?', default=None)
print(parser.parse_args())

Which works fine for test.py B C -a A and test.py -a A B C.

But when I do test.py B -a A C, it throws an error:

$ python3 test.py B -a A C
usage: test.py [-h] [-a A] b [c]
test.py: error: unrecognized arguments: C

So, how can I get it to accept the optional positional argument to be accepted even if there is a flag in between?

Note that this works if I remove the nargs='?', default=None, but then it's not optional. The problem also happens with nargs='*', but this doesn't happen for nargs=N (e.g. nargs=1, nargs=2) and doesn't happen for nargs='+'. nargs=argparse.REMAINDER makes it parse the flags as part of c (c = ['-a', 'A', 'C'], a = None)

Answer 1

This is a known issue, both here on SO and Python bug/issues, and doesn't have an easy fix. https://bugs.python.org/issue15112

It's the result of the basic parsing algorithm. This trys to parse positionals up to the next optional's flag. Then parse the flagged option (and however many arguments it needs). Then parse the next batch of positions, etc.

When the parser handles b, it can also handle c, even if there is just one string. c requires nothing. That means c gets 'used up' the first time it processes positionals.

In [50]: parser.parse_args(['one'])
Out[50]: Namespace(a=None, b='one', c=None)
In [51]: parser.parse_args(['one','two'])
Out[51]: Namespace(a=None, b='one', c='two')
In [52]: parser.parse_args(['one','-a','1','two'])
usage: ipython3 [-h] [-a A] b [c]
ipython3: error: unrecognized arguments: two
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2

/home/paul/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2971: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)
In [53]: parser.parse_known_args(['one','-a','1','two'])
Out[53]: (Namespace(a='1', b='one', c=None), ['two'])

With c used up (even though it just gets the default), there's nothing to consume the last string. It is an 'extra'.

parse_intermixed_args

Python 3.7 has added a parsing method that solves this issue, parse_intermixed_args. https://docs.python.org/3/library/argparse.html#intermixed-parsing

In [447]: import argparse37
In [448]: p = argparse37.ArgumentParser()
In [449]: p.add_argument('pos1');
In [450]: p.add_argument('-a');
In [451]: p.add_argument('pos2', nargs='?');

In [453]: p.parse_args('1 2 -a foo'.split())
Out[453]: Namespace(a='foo', pos1='1', pos2='2')

In [454]: p.parse_args('1 -a foo 2'.split())
usage: ipython3 [-h] [-a A] pos1 [pos2]
ipython3: error: unrecognized arguments: 2
...

In [455]: p.parse_intermixed_args('1 -a foo 2'.split())
Out[455]: Namespace(a='foo', pos1='1', pos2='2')

In [456]: p.parse_intermixed_args('1 2 -a foo'.split())
Out[456]: Namespace(a='foo', pos1='1', pos2='2')

It was added as a way of allowing a flagged Action in the middle of a '*' positional. But ends up working in this case with '?' Actions. Note the caution in the docs; it may not handle all argparse features.

In effect it deactivates the positionals, does a parse_known_args to get all optionals, and then parses the extras with just the positonals. See the code of parse_known_intermixed_args for details.