Python argparse AssertionError when number of arguments exceeds threshold

Question

I'm trying to write a Python script with a lot of arguments that I'd like to break up into clear groups using the argparse module as follows:

import argparse
parser = argparse.ArgumentParser(description='Placeholder text', add_help=False)
req = parser.add_argument_group('required arguments')
req.add_argument('-m','--mode', action='store', dest='mode', help='Operation mode', choices=['single', 'multi'], required=True, metavar='')
req.add_argument('-s','--snps', action='store', dest='snps', help='SNP BED file', required=True, metavar='')
req.add_argument('-r','--reads', action='store', dest='reads', help='Mapped reads file [sam or bam]', required=True, metavar='')
uni = parser.add_argument_group('universal optional arguments')
uni.add_argument('-p','--prefix', action='store', dest='prefix', help='Prefix for temp files and output [Default: TEST]', default='TEST', metavar='')
uni.add_argument('-b','--bam', action='store_true', dest='bam', help='Mapped read file type is bam (auto-detected if *.bam)')
uni.add_argument('-t','--single', action='store_true', dest='single', help='Mapped reads are single-end [Default: False]')
uni.add_argument('-n','--noclean', action='store_true', dest='noclean', help='Do not delete intermediate files (for debuging)')
uni.add_argument('-h', '--help', action='help', help='show this help message and exit')
mult = parser.add_argument_group('multi(plex) mode arguments')
mult.add_argument('-j','--jobs', action='store', dest='jobs', type=int, help='Divide into # of jobs [Default: 100]', default=100, metavar='')
mult.add_argument('-w','--walltime', action='store', dest='walltime', help='Walltime for each job [Default: 3:00:00]', default='3:00:00', metavar='')
mult.add_argument('-k','--mem', action='store', dest='memory', help='Memory for each job [Default: 5000MB]', default='5000MB', metavar='')
single = parser.add_argument_group('single mode arguments')
single.add_argument('-f','--suffix', action='store', dest='suff', help='Suffix for multiplexed files [set automatically]', default='', metavar='')

args = parser.parse_args()

However, when I run this script, with the '-h' option, instead of printing the help text, it spits out the following error:

Traceback (most recent call last):
  File "CountSNPLevelASE.py", line 61, in <module>
    args = parser.parse_args()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1688, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1720, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1926, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1866, in consume_optional
    take_action(action, args, option_string)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1794, in take_action
    action(self, namespace, argument_values, option_string)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 994, in __call__
    parser.print_help()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 2327, in print_help
    self._print_message(self.format_help(), file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 2301, in format_help
    return formatter.format_help()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 279, in format_help
    help = self._root_section.format_help()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 209, in format_help
    func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 330, in _format_usage
    assert ' '.join(opt_parts) == opt_usage
AssertionError

The odd thing is that if I comment out ANY of the non-required arguments, I get the proper help text output, like this:

usage: CountSNPLevelASE.py -m  -s  -r  [-b] [-t] [-n] [-h] [-j] [-w] [-k] [-f]

Placeholder text

required arguments:
  -m , --mode       Operation mode
  -s , --snps       SNP BED file
  -r , --reads      Mapped reads file [sam or bam]

universal optional arguments:
  -b, --bam         Mapped read file type is bam (auto-detected if *.bam)
  -t, --single      Mapped reads are single-end [Default: False]
  -n, --noclean     Do not delete intermediate files (for debuging)
  -h, --help        show this help message and exit

multi(plex) mode arguments:
  -j , --jobs       Divide into # of jobs [Default: 100]
  -w , --walltime   Walltime for each job [Default: 3:00:00]
  -k , --mem        Memory for each job [Default: 5000MB]

single mode arguments:
  -f , --suffix     Suffix for multiplexed files [set automatically]

I can't find a clear explanation for why this is happening. I've tried removing all whitespace to no avail. Any ideas? (I'm using Python 2.7.6 on Mac OS X 10.10.2 if that's helpful).

Answer 1

The problem lies in wrapping the usage. It displays fine when usage fits on one line, but fails when it splits it into 2.

There's a known bug issue involving this assertion. The usage formatting code is fragile, and can raise this assertion error when there are 'usual' characters in the usage, including and extra spaces.

It's the metavar='' of the 3 required arguments that's giving the problem. The optional optionals can have a '' metavar.

Notice the double spaces between these arguments in the short usage line?

...py -m  -s  -r  [-b]...

The assertion error is the result of those spaces not making it through the wrapping code intact.

If you can't find suitable alternative metavars for those 3 arguments, then we'll have to figure out how to modify your argparse code. Possibilities include commenting out the assertion statement, or trimming out those extra blanks from the one line usage. The big fix is a more robust usage formatter.