Find out which arguments were passed explicitly in argparse

Question

I need to handle two ways of configuring an application. One is via command line arguments, and the other one is from a JSON file (actually, the JSON file is the result of storing the arguments from a previous run).

I can handle this by merging the two namespace objects, something like this:

cli_args = some_parser.parse_args()
with open(json_file, 'r') as f:
    json_args = json.load(f)
all_args = argparse.Namespace()
all_args.__dict__ = {**vars(cli_args), **json_args}
# or all_args.__dict__ = {**json_args, **vars(cli_args)}

The problem is on the last line. If I choose the first version, arguments from the JSON file take precedence. If I choose the second version, arguments from the CLI take precedence.

I would like to have CLI arguments take precedence, but only if they were actually specified. The problem arises when the parser admits default values. In this case, the cli_args object will be populated with default values, and this will take precedence over JSON arguments.

For a simple example, let me take this parser:

parser = argparse.ArgumentParser()
parser.add_argument('--foo', default='FOO')
parser.add_argument('--bar', default='BAR')

Say I have a JSON file with

{
  "foo": "myfoo",
  "bar": "mybar"
}

and that I invoke my application with python myapp.py --foo hello.

I would like to get a namespace object having foo=hello, bar=mybar. Both ways of merging the arguments will give something different. First, if I give the JSON file precedence I will obtain foo=myfoo, bar=mybar. If I give the CLI the precedence, I get foo=hello, bar=BAR.

The problem is that I cannot see a way to distinguish which arguments in the namespace returned from parser.parse_args() were populated by the user, and which ones were filled in using default settings.

Is there a way to ask argparse which arguments were actually explicitly set on the command line, as opposed to being filled with defaults?

Answer 1

For the next person here looking for a solution to this problem, we can use Andrea's workaround and a sentinel to find out exactly which arguments were passed explicitly.

As Andrea pointed out, if we pass a namespace to parse_args(), then any existing names in the namespace override the defaults of the parser. So if we use a namespace that contains all the parameter names, then any that have changed their value after parsing must have been changed by explicit arguments.

We need to set everything in that namespace to something different than what the parser might use as a default anyway. The parser's defaults could well be None so that doesn't work; instead we need a sentinel object.

import argparse
from argparse import Namespace

# Create a sentinel.
# Could use sentinel = object() instead; this way makes it clear 
# if we ever print it that the object is a sentinel.
class _Sentinel:
    pass
sentinel = _Sentinel()

parser = argparse.ArgumentParser()
parser.add_argument('--foo', type=int, default=10)
args = parser.parse_args()

# Make a copy of args where everything is the sentinel.
sentinel_ns = Namespace(**{key:sentinel for key in vars(args)})
parser.parse_args(namespace=sentinel_ns)

# Now everything in sentinel_ns that is still the sentinel was not explicitly passed.
explicit = Namespace(**{key:(value is not sentinel)
                for key, value in vars(sentinel_ns).items()})

print("args.foo:", args.foo)
print("explicit.foo:", explicit.foo)

Running this script with no arguments will print

args.foo: 10
explicit.foo: False

And running this script with --foo 10 will print

args.foo: 10
explicit.foo: True

Answer 2

I could not find a way to ask argparse what arguments were explicitly set, but I found an alternative way to solve my issue.

Namely, the parse_args method accept a Namespace object that it will populate with the parsing results. Hence I can read the JSON content into a Namespace, then use parse_args() to add the arguments from command line. This will override the JSON settings, but only for explictly set arguments, not for defaults:

json_args = argparse.Namespace()
with open(json_file, 'r') as f:
    json_args.__dict__ = json.load(f)
all_args = parser.parse_args(namespace=json_args)