Which is the best way to allow configuration options be overridden at the command line in Python?

Question

I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.

Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.

I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.

Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...

Answer 1

I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:

Default with no user input:

$ ./argparse-partial.py Option is "default" 

Default from configuration file:

$ cat argparse-partial.config  [Defaults] option=Hello world! $ ./argparse-partial.py -c argparse-partial.config  Option is "Hello world!" 

Default from configuration file, overridden by commandline:

$ ./argparse-partial.py -c argparse-partial.config --option override Option is "override" 

argprase-partial.py follows. It is slightly complicated to handle -h for help properly.

import argparse import ConfigParser import sys  def main(argv=None):     # Do argv default this way, as doing it in the functional     # declaration sets it at compile time.     if argv is None:         argv = sys.argv      # Parse any conf_file specification     # We make this parser with add_help=False so that     # it doesn't parse -h and print help.     conf_parser = argparse.ArgumentParser(         description=__doc__, # printed with -h/--help         # Don't mess with format of description         formatter_class=argparse.RawDescriptionHelpFormatter,         # Turn off help, so we print all options in response to -h         add_help=False         )     conf_parser.add_argument("-c", "--conf_file",                         help="Specify config file", metavar="FILE")     args, remaining_argv = conf_parser.parse_known_args()      defaults = { "option":"default" }      if args.conf_file:         config = ConfigParser.SafeConfigParser()         config.read([args.conf_file])         defaults.update(dict(config.items("Defaults")))      # Parse rest of arguments     # Don't suppress add_help here so it will handle -h     parser = argparse.ArgumentParser(         # Inherit options from config_parser         parents=[conf_parser]         )     parser.set_defaults(**defaults)     parser.add_argument("--option")     args = parser.parse_args(remaining_argv)     print "Option is \"{}\"".format(args.option)     return(0)  if __name__ == "__main__":     sys.exit(main())