Why isn't fromfile-prefix-chars in Python argparse working?

Question

I'm trying to use the fromfile-prefix-chars feature of argparse in Python to load all my command line arguments from a file, but it keeps complaining that I haven't specified some argument.

The code:

import argparse

def go():
   parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
   parser.add_argument("--option1")
   parser.add_argument("--option2", type=int, required=True)
   args = parser.parse_args()

if __name__ == "__main__":
    go()

The argument file:

--option1 foo
--option2 1234

The command line and output:

$ python testargparse.py @testargs
usage: testargparse.py [-h] [--option1 OPTION1] --option2 OPTION2
testargparse.py: error: argument --option2 is required

You can see that I'm providing the required argument in the file, but argparse isn't seeing it.

Answer 1

From the documentation:

Arguments read from a file must by default be one per line ... and are treated as if they were in the same place as the original file referencing argument on the command line. So in the example above, the expression ['-f', 'foo', '@args.txt'] is considered equivalent to the expression ['-f', 'foo', '-f', 'bar'].

In the example:

fp.write('-f\nbar')

So the file contains:

-f
bar

In other words, each of the file lines corresponds to one 'word' (blank separated) in the command line. --option1=foo is one word. --option1 foo is interpreted just as though it was quoted in the command line,eg. prog.py '--option1 foo' '--option2 1234'

The https://docs.python.org/dev/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args has a custom function that will split lines on spaces. Experiment with that if you want to stick with the argument file.

---

import argparse

with open('args.txt', 'w') as fp:
    fp.write('--option1 foo\n--option2 1234')  # error
    # but works with modifed 'convert...'
    #fp.write('--option1=foo\n--option2=1234')  # works
    #fp.write('--option1\nfoo\n--option2\n1234') # works

def convert_arg_line_to_args(arg_line):
    for arg in arg_line.split():
        if not arg.strip():
            continue
        yield arg
"""
default line converter:
def convert_arg_line_to_args(self, arg_line):
    return [arg_line]
"""

def go():
   parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
   parser.convert_arg_line_to_args = convert_arg_line_to_args
   parser.add_argument("--option1")
   parser.add_argument("--option2", type=int, required=True)
   args = parser.parse_args(['@args.txt'])
   print args

if __name__ == "__main__":
    go()

---

@toes suggested using shlex to parse the file. shlex has a nice feature in that it strips of unnecessary quotes.

shlex can be used to split individual lines of the file.

def sh_split(arg_line):
    for arg in shlex.split(arg_line):
        yield arg
parser.convert_arg_line_to_args = sh_split

Or it can replace the whole @file read method (_read_args_from_files)- this should function the same as @toes answer, except that the @file string can be anywhere in the commandline (or even be repeated).

def at_read_fn(arg_strings):
    # expand arguments referencing files
    new_arg_strings = []
    for arg_string in arg_strings:
        if not arg_string or not arg_string.startswith('@'):
            new_arg_strings.append(arg_string)
        else:
            with open(arg_string[1:]) as args_file:
                arg_strings = shlex.split(args_file.read())
                new_arg_strings.extend(arg_strings)
    return new_arg_strings
parser._read_args_from_files = at_read_fn

Obviously a cleaner production version would modify these methods in an ArgumentParser subclass.

Answer 2

The problem is that, when specified in a file, each argument must have an '=' between it and the option name. While argparse is somewhat more flexible on that format when run from the command line (where space or = is ok), when run from the file it must have an '='.

So, a working argument file would be:

--option1=foo
--option2=1234

Something else to be aware of, be sure you don't have any extra whitespace at the end of the lines or that whitespace will get included with the option when argparse reads the file.