Python argparse REMAINDER is not clear

Question

As documentation suggests:

argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:

>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print parser.parse_args('--foo B cmd --arg1 XX ZZ'.split())
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

I tried to use this to exactly the same purpose, but in some circumstances it seems buggy for me (or perhaps I get the concept wrong):

import argparse

a = argparse.ArgumentParser()

a.add_argument('-qa', nargs='?')
a.add_argument('-qb', nargs='?')
a.add_argument('rest', nargs=argparse.REMAINDER)

a.parse_args('-qa test ./otherutil bar -q atr'.split())

Result:

test.py: error: ambiguous option: -q could match -qa, -qb

So apparently, if the otherutil has such arguments which somehow "collide" with the arguments given to argparse, it doesn't seem to work correctly.

I would expect when argparse reaches the REMAINDER kind of argument, it just uses up all the strings in the end of the list without any further parsing. Can I reach this effect somehow?

Answer 1

I ran into this while trying to dispatch options to an underlying utility. The solution I wound up using was nargs='*' instead of nargs=argparse.REMAINDER, and then just use the "pseudo-argument" -- to separate the options for my command and the underlying tool:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--myflag', action='store_true')
>>> parser.add_argument('toolopts', nargs='*')
>>> parser.parse_args('--myflag -- -a --help'.split())
Namespace(myflag=True, toolopts=['-a', '--help'])

This is reasonably easy to document in the help output.

Answer 2

This has more to do with handling of abbreviations than with the REMAINDER nargs.

In [111]: import argparse                                                                 
In [112]: a = argparse.ArgumentParser() 
     ...:  
     ...: a.add_argument('-qa', nargs='?') 
     ...: a.add_argument('-qb', nargs='?')                                                

In [113]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
usage: ipython3 [-h] [-qa [QA]] [-qb [QB]]
ipython3: error: ambiguous option: -q could match -qa, -qb

argparse does a 2 pass parsing. First it tries to categorize the strings as options (flags) or arguments. Second it alternates between parsing positionals and optionals, allocating arguments according to the nargs.

Here the ambiguity occurs in the first pass. It's trying to match '-q' with the two available optionals. REMAINDER's special action (absorbing '-q' as though it were an plain string) doesn't occur until the second pass.

Newer argparse versions allow us to turn off the abbreviation handling:

In [114]: a.allow_abbrev                                                                  
Out[114]: True
In [115]: a.allow_abbrev=False                                                            
In [116]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
usage: ipython3 [-h] [-qa [QA]] [-qb [QB]]
ipython3: error: unrecognized arguments: ./otherutil bar -q atr

And if I add the REMAINDER action:

In [117]: a.add_argument('rest', nargs=argparse.REMAINDER) 

In [118]: a.parse_args('-qa test ./otherutil bar -q atr'.split())                         
Out[118]: Namespace(qa='test', qb=None, rest=['./otherutil', 'bar', '-q', 'atr'])

The use of '--' as @Colin suggests works because that string is recognized in the first pass:

In [119]: a.allow_abbrev=True                                                             
In [120]: Out[117].nargs='*'                                                              
In [121]: a.parse_args('-qa test -- ./otherutil bar -q atr'.split())                      
Out[121]: Namespace(qa='test', qb=None, rest=['./otherutil', 'bar', '-q', 'atr'])