argparse subcommands with nested namespaces

Question

Does argparse provide built-in facilities for having it parse groups or parsers into their own namespaces? I feel like I must be missing an option somewhere.

Edit: This example is probably not exactly what I should be doing to structure the parser to meet my goal, but it was what I worked out so far. My specific goal is to be able to give subparsers groups of options that are parsed into namespace fields. The idea I had with parent was simply to use common options for this same purpose.

Example:

import argparse

# Main parser
main_parser = argparse.ArgumentParser()
main_parser.add_argument("-common")

# filter parser
filter_parser = argparse.ArgumentParser(add_help=False)
filter_parser.add_argument("-filter1")
filter_parser.add_argument("-filter2")

# sub commands
subparsers = main_parser.add_subparsers(help='sub-command help')

parser_a = subparsers.add_parser('command_a', help="command_a help", parents=[filter_parser])
parser_a.add_argument("-foo")
parser_a.add_argument("-bar")

parser_b = subparsers.add_parser('command_b', help="command_b help", parents=[filter_parser])
parser_b.add_argument("-biz")
parser_b.add_argument("-baz")

# parse
namespace = main_parser.parse_args()
print namespace

This is what I get, obviously:

$ python test.py command_a -foo bar -filter1 val
Namespace(bar=None, common=None, filter1='val', filter2=None, foo='bar')

But this is what I am really after:

Namespace(bar=None, common=None, foo='bar', 
          filter=Namespace(filter1='val', filter2=None))

And then even more groups of options already parsed into namespaces:

Namespace(common=None, 
          foo='bar', bar=None,  
          filter=Namespace(filter1='val', filter2=None),
          anotherGroup=Namespace(bazers='val'),
          anotherGroup2=Namespace(fooers='val'),
          )

I've found a related question here but it involves some custom parsing and seems to only covers a really specific circumstance.

Is there an option somewhere to tell argparse to parse certain groups into namespaced fields?

Answer 1

In this script I have modified the __call__ method of the argparse._SubParsersAction. Instead of passing the namespace on to the subparser, it passes a new one. It then adds that to the main namespace. I only change 3 lines of __call__.

import argparse

def mycall(self, parser, namespace, values, option_string=None):
    parser_name = values[0]
    arg_strings = values[1:]

    # set the parser name if requested
    if self.dest is not argparse.SUPPRESS:
        setattr(namespace, self.dest, parser_name)

    # select the parser
    try:
        parser = self._name_parser_map[parser_name]
    except KeyError:
        args = {'parser_name': parser_name,
                'choices': ', '.join(self._name_parser_map)}
        msg = _('unknown parser %(parser_name)r (choices: %(choices)s)') % args
        raise argparse.ArgumentError(self, msg)

    # CHANGES
    # parse all the remaining options into a new namespace
    # store any unrecognized options on the main namespace, so that the top
    # level parser can decide what to do with them
    newspace = argparse.Namespace()
    newspace, arg_strings = parser.parse_known_args(arg_strings, newspace)
    setattr(namespace, 'subspace', newspace) # is there a better 'dest'?

    if arg_strings:
        vars(namespace).setdefault(argparse._UNRECOGNIZED_ARGS_ATTR, [])
        getattr(namespace, argparse._UNRECOGNIZED_ARGS_ATTR).extend(arg_strings)

argparse._SubParsersAction.__call__ = mycall

# Main parser
main_parser = argparse.ArgumentParser()
main_parser.add_argument("--common")

# sub commands
subparsers = main_parser.add_subparsers(dest='command')

parser_a = subparsers.add_parser('command_a')
parser_a.add_argument("--foo")
parser_a.add_argument("--bar")

parser_b = subparsers.add_parser('command_b')
parser_b.add_argument("--biz")
parser_b.add_argument("--baz")

# parse
input = 'command_a --foo bar --bar val --filter extra'.split()
namespace = main_parser.parse_known_args(input)
print namespace

input = '--common test command_b --biz bar --baz val'.split()
namespace = main_parser.parse_args(input)
print namespace

This produces:

(Namespace(command='command_a', common=None, 
    subspace=Namespace(bar='val', foo='bar')), 
['--filter', 'extra'])

Namespace(command='command_b', common='test', 
    subspace=Namespace(baz='val', biz='bar'))

I used parse_known_args to test how extra strings are passed back to the main parser.

I dropped the parents stuff because it does not add anything to this namespace change. it is just a convenient way of defining a set of arguments that several subparsers use. argparse does not keep a record of which arguments were added via parents, and which were added directly. It is not a grouping tool

argument_groups don't help much either. They are used by the Help formatter, but not by parse_args.

I could subclass _SubParsersAction (instead of reassigning __call__), but then I'd have change the main_parse.register.

Answer 2

I'm not entirely sure what you're asking, but I think what you want is for an argument group or sub-command to put its arguments into a sub-namespace.

As far as I know, argparse does not do this out of the box. But it really isn't hard to do by postprocessing the result, as long as you're willing to dig under the covers a bit. (I'm guessing it's even easier to do it by subclassing ArgumentParser, but you explicitly said you don't want to do that, so I didn't try that.)

parser = argparse.ArgumentParser()
parser.add_argument('--foo')
breakfast = parser.add_argument_group('breakfast')
breakfast.add_argument('--spam')
breakfast.add_argument('--eggs')
args = parser.parse_args()

Now, the list of all destinations for breakfast options is:

[action.dest for action in breakfast._group_actions]

And the key-value pairs in args is:

args._get_kwargs()

So, all we have to to is move the ones that match. It'll be a little easier if we construct dictionaries to create the namespaces from:

breakfast_options = [action.dest for action in breakfast._group_actions]
top_names = {name: value for (name, value) in args._get_kwargs()
             if name not in breakfast_options}
breakfast_names = {name: value for (name, value) in args._get_kwargs()
                   if name in breakfast_options}
top_names['breakfast'] = argparse.Namespace(**breakfast_names)
top_namespace = argparse.Namespace(**top_names)

And that's it; top_namespace looks like:

Namespace(breakfast=Namespace(eggs=None, spam='7'), foo='bar')

---

Of course in this case, we've got one static group. What if you wanted a more general solution? Easy. parser._action_groups is a list of all groups, but the first two are the global positional and keyword groups. So, just iterate over parser._action_groups[2:], and do the same thing for each that you did for breakfast above.

---

What about sub-commands instead of groups? Similar, but the details are different. If you've kept around each subparser object, it's just whole other ArgumentParser. If not, but you did keep the subparsers object, it's a special type of Action, whose choices is a dict whose keys are the subparser names and whose values are the subparsers themselves. If you kept neither… start at parser._subparsers and figure it out from there.

At any rate, once you know how to find the names you want to move and where you want to move them, it's the same as with groups.

---

If you've got, in addition to global args and/or groups and subparser-specific args and/or groups, some groups that are shared by multiple subparsers… then conceptually it gets tricky, because each subparser ends up with references to the same group, and you can't move it to al of them. But fortunately, you're only dealing with exactly one subparser (or none), so you can just ignore the other subparsers and move any shared group under the selected subparser (and any group that doesn't exist in the selected subparser, either leave at the top, or throw away, or pick one subparser arbitrarily).