Create variable key/value pairs with argparse (python)

Question

I'm using argparse module to set my command line options. I'm also using a dict as a config in my application. Simple key/value store.

What I'm looking for is a possibility to override JSON options using command line arguments, without defining all possible arguments in advance. Something like --conf-key-1 value1 --conf-key-2 value2, which would create a dict {'key_1': 'value1','key_2': 'value2'} ('-' in the argument is replaced by '_' in the dict). Then I can combine this dict with my JSON config (dict).

So basically I would like to define --conf-* as an argument, where * can be any key and what comes after is the value.

I did find configargparse module, but as far as I can see I start with a dict I already use.

Any ideas how I could approach this?

Answer 1

I had a similar issue and found a very workable pattern that works well with argparse (here three key-pairs: foo, bar and baz:

mycommand par1 --set foo=hello bar="hello world" baz=5

1. Defining the optional, multivalued argument

The set argument must be defined so:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence (nargs='+').

The result is a list of strings e.g. ["foo=hello", "bar=hello world", "baz=5"] in args.set, which we now need to parse (note how the shell has processed and removed the quotes!).

2. Parsing the result

For this we need 2 helper functions:

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

At this point it is very simple:

# parse the key-value pairs
values = parse_vars(args.set)

You now have a dictionary:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note how the values are always returned as strings.

This method is also documented as a git gist.

Answer 2

To simplify slightly on fralaus answer, the 2 methods can be combined into one easily.

Note: My docstring etc differ as I was using it for ansible extra_vars, but the core logic for string splitting came from fralaus' answer.

 def parse_vars(extra_vars):
     """
     Take a list of comma seperated key value pair strings, seperated
     by comma strings like 'foo=bar' and return as dict.
     :param extra_vars: list[str] ['foo=bar, 'key2=value2']

     :return: dict[str, str] {'foo': 'bar', 'key2': 'value2'}
     """
     vars_list = []
     if extra_vars:
         for i in extra_vars:
            items = i.split('=')
            key = items[0].strip()
            if len(items) > 1:
                value = '='.join(items[1:])
                vars_list.append((key, value))
     return dict(vars_list)

print parse_vars(args.set)

 $ test.py --set blah=gar one=too
>> {"blah": "gar", "one": "too"}