Running tests against code examples in a README.md?

Question

Does anyone know of an open-source project or program for running tests against code examples in README.md?

A perennial problem, my documentation tends to drift out of date with the code. For example, a code snippet in the README.md will no longer work with the current version, and isn't caught until a new developer on-boards onto the project. Is it possible to include README.md code snippets in my test suite?

For example, the usage of say.nancat with sample params:

# $ node

> const say = require('say')
> say.nancat('grumpy is best')
'grumpy is best'

The program would initialize an environment with the '#' (not shown in README.md because the context is assumed), run the '>' line and pass/fail based on the next line. Simular to doctests in python.

Loads of people have the problem of keeping README.md and other docs current with the code, so I was hoping there was an off-the-shelf solution. I've looked (DuckDuckGo) to no avail.

Answer 1

May be byexample is what you are looking for.

It is a tool to run the snippets of code (aka examples) in a text file and check their outputs. It is like Python's doctests but it works for Javascript, Ruby, Python and others (even for C and C++).

The Javascript examples can be written in a README.md like:

```javascript
1 + 2

out:
3
```

or like:

```javascript
> 1 + 2
3
```

Then, you run them from the command line:

$ byexample -l javascript README.md
[PASS] Pass: 2 Fail: 0 Skip: 0

And that's it. The full documentation of the tool can be found here and here, and the particular comments and limitations for Javascript is here.

Disclaimer: I'm the author of byexample and I created it for the same reason that rmharrison wrote in his question.

Like him, my documentation was "out of sync" in time to time and the only way to notice that was running the examples by hand. For that reason I created this tool to automatically check and validate the docs.

It is really useful to me; I really hope that it would be useful to others.