how to get argparse to read arguments from a file with an option rather than prefix

Question

I would like to know how to use python's argparse module to read arguments both from the command line and possibly from text files. I know of argparse's fromfile_prefix_chars but that's not exactly what I want. I want the behavior, but I don't want the syntax. I want an interface that looks like this:

$ python myprogram.py --foo 1 -A somefile.txt --bar 2

When argparse sees -A, it should stop reading from sys.argv or whatever I give it, and call a function I write that will read somefile.text and return a list of arguments. When the file is exhausted it should resume parsing sys.argv or whatever. It's important that the processing of the arguments in the file happen in order (ie: -foo should be processed, then the arguments in the file, then -bar, so that the arguments in the file may override --foo, and --bar might override what's in the file).

Is such a thing possible? Can I write a custom function that pushes new arguments onto argparse's stack, or something to that effect?

Answer 1

You can solve this by using a custom argparse.Action that opens the file, parses the file contents and then adds the arguments then.

For example this would be a very simple action:

class LoadFromFile (argparse.Action):
    def __call__ (self, parser, namespace, values, option_string = None):
        with values as f:
            # parse arguments in the file and store them in the target namespace
            parser.parse_args(f.read().split(), namespace)

Which you can the use like this:

parser = argparse.ArgumentParser()
# other arguments
parser.add_argument('--file', type=open, action=LoadFromFile)
args = parser.parse_args()

The resulting namespace in args will then also contain any configuration that was also loaded from the file where the file contained arguments in the same syntax as on the command line (e.g. --foo 1 --bar 2).

If you need a more sophisticated parsing, you can also parse the in-file configuration separately first and then selectively choose which values should be taken over. For example, since the arguments are evalutated in the order they are specified, it might make sense to prevent the configurations in the file from overwriting values that have been explicitly specified ont the command line. This would allow using the configuration file for defaults:

def __call__ (self, parser, namespace, values, option_string=None):
    with values as f:
        contents = f.read()

    # parse arguments in the file and store them in a blank namespace
    data = parser.parse_args(contents.split(), namespace=None)
    for k, v in vars(data).items():
        # set arguments in the target namespace if they haven’t been set yet
        if getattr(namespace, k, None) is not None:
            setattr(namespace, k, v)

Of course, you could also make the file reading a bit more complicated, for example read from JSON first.

Answer 2

You commented that

I need to be able to write my own function to read that file and return the arguments (it's not in a one-argument-per-line format) –

There is a provision in the existing prefix-file handler to change how the file is read. The file is read by a 'private' method, parser._read_args_from_files, but it calls a simple public method that converts a line to strings, default one-argument-per-line action:

def convert_arg_line_to_args(self, arg_line):
    return [arg_line]

It was written this way so you could easily customize it. https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args

A useful override of this method is one that treats each space-separated word as an argument:

def convert_arg_line_to_args(self, arg_line):
    for arg in arg_line.split():
        if not arg.strip():
            continue
        yield arg

In the test_argparse.py unittesting file there is a test case for this alternative.

---

But if you still want to trigger this read with an argument option, instead of a prefix character, then the custom Action approach is a good one.

You could though write your own function that processes argv before it is passed to the parser. It could be modeled on parser._read_args_from_files.

So you could write a function like:

def read_my_file(argv):
    # if there is a '-A' string in argv, replace it, and the following filename
    # with the contents of the file (as strings)
    # you can adapt code from _read_args_from_files
    new_argv = []
    for a in argv:
        ....
        # details left to user
    return new_argv

Then invoke your parser with:

parser.parse_args(read_my_file(sys.argv[1:]))

And yes, this could be wrapped in a ArgumentParser subclass.