Why does argparse include default value for optional argument even when argument is specified?

Question

I'm using argparse with Python 3.6. I am using optional arguments to collect my program parameters. For some of these, I have reasonable default values so I configure the parser with the default for that argument.

In [2]: import argparse
   ...: import shlex
   ...: 
   ...: parser = argparse.ArgumentParser()
   ...: parser.add_argument('-s', '--samples', action='store', nargs='+', type=int, required=True,
   ...:                     help='number of samples')
   ...: parser.add_argument('-r', '--regions', action='append', nargs='+', type=str, default=['all'],
   ...:                     help='one or more region names. default to [\'all\']')

When not specifying the -r/--regions argument, I expect to see the configured default value (and I do).

In [3]: s = '-s 37'
   ...: parser.parse_args(shlex.split(s))
Out[3]: Namespace(regions=['all'], samples=[37])

When specifying the -r/--regions argument, I expect to see only the values I provide with the argument, but the default shows up as well.

In [5]: s = '-s 37 -r foo'
...: parser.parse_args(shlex.split(s))
Out[5]: Namespace(regions=['all', ['foo']], samples=[37])

This is not what I expected. I would expect the default to be present only when the optional argument is not present. I stepped through the argparse code. I can't find where the default value is included. Based on the comments, it appears the logic is to add the default value to the resulting Namespace before processing the actual argument value supplied. I would have expected it to be the reverse (i.e. apply the default only when you get to the end of the args and have not seen an argument which has a default.

Can anyone shed light on this? Am I incorrectly using or understanding the purpose of the default option on an optional argument? Is there a way to achieve the behavior I'm seeking (i.e. if the optional is not provided, use the default value in the Namespace)?

Answer 1

The logic for handling defaults is to insert all defaults into the namespace at the start of parsing. Then let parsing replace them. Then at the end of parsing there's a complicated piece of logic:

for each value in the namespace
   if it is a string and equals the default
      evaluate the string (with `type`) and put it back

For ordinary store actions this works fine, and lets you provide defaults as strings or any value of your choice.

With append this produces your unexpected value. It puts the ['all'] on the namespace, and then appends the new values to that. Since your nargs is '+', it appends a list, resulting in that mix of string and lists.

The append action cannot tell whether it is appending the new value to a list provided by default or a list that's the result of several previous appends. With a None default it will create an empty list, and append to that.

While this doesn't perform as you expected, it actually gives you a lot of control.

The simplest way around this is to leave the default as None. After parsing, just check if this attribute is None, and if so, replace it with your ['all']. It is not evil or contrary to the design intentions of argparse developers. Somethings are easier after all the input has been parsed.

The issue has been raised on Python bug/issues, http://bugs.python.org/issue16399, and probably here on SO before. But I suspect the best a patch can do is to add a note to the documentation, similar to this one from optparse:

"The append action calls the append method on the current value of the option. This means that any default value specified must have an append method. It also means that if the default value is non-empty, the default elements will be present in the parsed value for the option, with any values from the command line appended after those default values".

See the bug/issue for ideas about writing your own append Action subclass.