Python argparse: Combine optional parameters with nargs=argparse.REMAINDER

Question

I must be missing something obvious. The goal is to use argparse with the first parameter required, a second optional and any other remaining parameters optional.

To show the issue I made two test parsers; the only difference between them is using nargs=argparse.REMAINDER in one and nargs='*' in the other.

def doParser1(argsin):
    parser = argparse.ArgumentParser(description='Parser demo.')
    parser.add_argument('req1', help='first required parameter')
    parser.add_argument('--opt1', help='first optional parameter')
    parser.add_argument('leftovers', nargs=argparse.REMAINDER,
                    help='all the other parameters')
    argsout = parser.parse_args(args=argsin)
    print argsout
    return argsout

def doParser2(argsin):
    parser = argparse.ArgumentParser(description='Parser demo.')
    parser.add_argument('req1', help='first required parameter')
    parser.add_argument('--opt1', help='first optional parameter')
    parser.add_argument('leftovers', nargs='*',
                    help='all the other parameters')
    argsout = parser.parse_args(args=argsin)
    print argsout
    return argsout

If there are no extra parameters, parser2 works. This is the input followed by parser1 and parser 1:

input: ['req1value', '--opt1', 'opt1value']
Namespace(leftovers=['--opt1', 'opt1value'], opt1=None, req1='req1value')
Namespace(leftovers=None, opt1='opt1value', req1='req1value')

If there are extra parameters, the opt1 value is missed in parser1 and parser2 just gets confused:

input: ['req1value', '--opt1', 'opt1value', 'r1', 'r2']
Namespace(leftovers=['--opt1', 'opt1value', 'r1', 'r2'], opt1=None, req1='req1value')
usage: py-argparse.py [-h] [--opt1 OPT1]
                  [-leftovers [LEFTOVERS [LEFTOVERS ...]]]
                  req1
py-argparse.py: error: unrecognized arguments: r1 r2

The expected output should be:

Namespace(leftovers=['r1', 'r2'], opt1='opt1value', req1='req1value')

It seems this should be a simple case and what is here is simplified from what I'm really trying to do. I've tried making leftovers optional, adding a variety of other options, but nothing works any better.

Any help would be appreciated.

Answer 1

You could use parse_known_args:

import argparse
parser = argparse.ArgumentParser(description='Parser demo.')
parser.add_argument('req1', help='first required parameter')
parser.add_argument('--opt1', help='first optional parameter')

args, leftovers = parser.parse_known_args(['req1value', '--opt1', 'opt1value'])
print(args, leftovers)
# (Namespace(opt1='opt1value', req1='req1value'), [])

args, leftovers = parser.parse_known_args(['req1value', '--opt1', 'opt1value', 'r1', 'r2'])
print(args, leftovers)
# (Namespace(opt1='opt1value', req1='req1value'), ['r1', 'r2'])

Answer 2

--opt1 needs to come before "unnamed" arguments. Your real test cases should be:

    ['--opt1', 'opt1value', 'req1value']

and

    ['--opt1', 'opt1value', 'req1value', 'r1', 'r2']

Answer 3

The inter mixing of positionals and optionals is tricky when one or more of the positionals is of the 'zero or more' type (? * REMAINDER). The simple solution is to not mix them - give the optionals first, then all the positionals afterwards.

Here's what's going on:

input: ['req1value', '--opt1', 'opt1value']
Namespace(leftovers=['--opt1', 'opt1value'], opt1=None, req1='req1value')

Because of the req1value string the parser first parses for positionals. req1 wants 1 string, leftovers grabs everything else including --opt1.

Namespace(leftovers=None, opt1='opt1value', req1='req1value')

With * leftovers is satisfied with [], no string, hence None (actually I get []). --opt1 is parsed as an optional.

input: ['req1value', '--opt1', 'opt1value', 'r1', 'r2']
...
py-argparse.py: error: unrecognized arguments: r1 r2

As before * leftovers is set to []. -opt1 is processed. But now there are 2 strings with no place to put them. You intended them to go into leftovers, but that was already used. If leftovers was + it would have taken them as you intended.

The key is that when it tries to parse the 1st positional, it also tries to parse all the positionals that it can. At one level parse_args is doing re.match('(A)(A*)','AOA') producing groups ('A', '').

There are 2 proposed patches that deal with this issue. One uses the 2 step parse_known_args to allow a complete mixing of optionals and positionals. This is the kind of behavior that users of optparse might have come to expect.

The other patch tries to delay the handling of positionals that can accept 0 argument strings http://bugs.python.org/issue15112 .