Extracting "extra" docstrings from Python code?

Question

Python docstrings that immediately follows the declaration of a class or function are placed in the __doc__ attribute.

The Question: How does one extract additional "internal" docstrings that occur later on in a function?

Update: Such literal statements are elided by the compiler. Could I perhaps get to them (and their line number) via the AST?

---

Why do I ask?

I've had a (not fully baked) idea to use such "internal" docstrings to delineate Given/When/Then sections of an Agile Scenario:

def test_adding():
    """Scenario: Adding two numbers"""
    adder = Adder()
    """When I add 2 and 3"""
    result = adder.add(2, 3)
    """Then the result is 5"""
    assert result == 5

By extracting the docstrings, the test-running framework could generate output like this:

Scenario: Adding two numbers
   When I add 2 and 3 (PASS)
   Then the result is 5 (FAIL)

AssertionError   Traceback
...

I think this would be more concise than the approach taken in Behave, Freshen, Lettuce, PyCukes, which require defining a separate function for each step. I don't like having to repeat the text of the step as a function name (@When("I add numbers") def add_numbers()). But unlike a plain unit test, the docstrings will add the ability to print out a business-readable scenario for reference.

Answer 1

You could parse your tests using the ast module, and manually walk the tree and setup tests, etc. There are probably better ways of doing this (you could use ast.NodeVisitor or ast.NodeTransfomer and the visitor pattern perhaps), but here's an example:

import ast, inspect

def find_tests(module):
    # generate AST from module's source
    tree = ast.parse(inspect.getsource(module))
    # return tests in module, assuming they are top level function definitions
    return [node for node in tree.body if isinstance(node, ast.FunctionDef)]

def print_docstrings(test):
    for node in test.body:
        if isinstance(node, ast.Expr):
            # print lineno and docstring
            print node.value.lineno, node.value.s

if __name__ == '__main__':
    import test_adding
    for test in find_tests(test_adding):
        print_docstrings(test)

You might also be interested in konira.

Answer 2

You can't, since the compiler elides literal statements.

>>> def foo():
...   'docstring'
...   3
...   'bar'
... 
>>> dis.dis(foo)
  4           0 LOAD_CONST               1 (None)
              3 RETURN_VALUE