Manuel

Manuel lets you mix and match traditional doctests with custom test syntax.

Several plug-ins are included that provide new test syntax (see Included Functionality). You can also create your own plug-ins.

For example, if you’ve ever wanted to include a large chunk of Python in a doctest but were irritated by all the “>>>” and ”...” prompts required, you’d like the manuel.codeblock module. It lets you execute code using Sphinx-style ”.. code-block:: python” directives. The markup looks like this:

.. code-block:: python

    import foo

    def my_func(bar):
        return foo.baz(bar)

Incidentally, the implementation of manuel.codeblock is only 23 lines of code.

The plug-ins included in Manuel make good examples while being quite useful in their own right. The Manuel documentation makes extensive use of them as well. Follow the “Show Source” link to the left to see the reST source of this document.

For a large example of creating test syntax, take a look at the FIT Table Example or for all the details, Theory of Operation.

To see how to get Manuel wired up see Getting Started.

Included Functionality

Manuel includes several plug-ins out of the box:

manuel.capture
stores regions of a document in variables for later processing
manuel.codeblock
executes code in ”.. code-block:: python” blocks
manuel.doctest
provides traditional doctest processing as a Manuel plug-in
manuel.footnote
executes code in reST-style footnodes each time they’re referenced (good for getting incidental code out of the main flow of a document)
manuel.ignore
ignores parts of a document while running tests
manuel.isolation
makes it easier to have test isolation in doctests
manuel.testcase
identify parts of tests as individual test cases so they can be run independently

Getting Started

The plug-ins used for a test are composed together using the “+” operator. Let’s say you wanted a test that used doctest syntax as well as footnotes. You would create a Manuel instance to use like this:

import manuel.doctest
import manuel.footnote

m = manuel.doctest.Manuel()
m += manuel.footnote.Manuel()

You would then pass the Manuel instance to a manuel.testing.TestSuite, including the names of documents you want to process:

manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')

Using unittest

The simplest way to get started with Manuel is to use unittest to run your tests:

import manuel.codeblock
import manuel.doctest
import manuel.testing
import unittest

def test_suite():
    m = manuel.doctest.Manuel()
    m += manuel.codeblock.Manuel()
    return manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')

if __name__ == '__main__':
    unittest.TextTestRunner().run(test_suite())

Using zope.testing

If you want to use a more featureful test runner you can use zope.testing’s test runner (usable stand-alone – it isn’t dependent on the Zope application server). Create a file named tests.py with a test_suite() function that returns a test suite.

The suite can be either a manuel.testing.TestSuite object or a unittest.TestSuite as demonstrated below.

import manuel.codeblock
import manuel.doctest
import manuel.testing

def test_suite():
    suite = unittest.TestSuite()

    # here you add your other tests to the suite...

    # now you can add the Manuel tests
    m = manuel.doctest.Manuel()
    m += manuel.codeblock.Manuel()
    suite.addTest(manuel.testing.TestSuite(m,
        'test-one.txt', 'test-two.txt'))

    return suite

Others

If you know out how to make Manuel work with other test runners (nose, py.test, etc.), please send me an email and I’ll expand this section.

Customizing the TestCase class

Manuel has its own manuel.testing.TestClass class that manuel.testing.TestSuite uses. If you want to customize it, you can pass in your own class to TestSuite.

 import os.path
 import manuel.testing

 class StripDirsTestCase(manuel.testing.TestCase):
     def shortDescription(self):
         return os.path.basename(str(self))
 suite = manuel.testing.TestSuite(
     m, path_to_test, TestCase=StripDirsTestCase)

>>> list(suite)[0].shortDescription()
'bugs.txt'

Doctests

Manuel is all about making testable documents and well-documented tests. Of course, Python’s doctest module is a long-standing fixture in that space, so it only makes sense for Manuel to support doctest syntax.

Handling doctests is easy:

import manuel.doctest

m = manuel.doctest.Manuel()
suite = manuel.testing.TestSuite(m, 'my-doctest.txt')

Of course you can mix in other Manuel syntax plug-ins as well (including ones you write yourself).

import manuel.doctest
import manuel.codeblock

m = manuel.doctest.Manuel()
m += manuel.codeblock.Manuel()
suite = manuel.testing.TestSuite(m, 'my-doctest-with-code-blocks.txt')

The manuel.doctest.Manuel constructor also takes optionflags and checker arguments.

m = manuel.doctest.Manuel(optionflags=optionflags, checker=checker)

See the doctest documentation for more information about the available options and output checkers

Note

zope.testing.renormalizing provides an OutputChecker for smoothing out differences between actual and expected output for things that are hard to control (like memory addresses and time). See the module’s doctests for more information on how it works. Here’s a short example that smoothes over the differences between CPython’s and PyPy’s NameError messages:

import re
import zope.testing.renormalizing
checker = zope.testing.renormalizing.RENormalizing([
    (re.compile(r"NameError: global name '([a-zA-Z0-9_]+)' is not defined"),
     r"NameError: name '\1' is not defined"),
])

Capturing Blocks

When writing documentation the need often arises to describe the contents of files or other non-Python information. You may also want to put that information under test. manuel.capture helps with that.

For example, if you were writing the problems for a programming contest, you might want to describe the input and output files for each challenge, but you want to be sure that your examples are correct.

To do that you might write your document like this:

Challenge 1
===========

Write a program that sorts the numbers in a file.


Example
-------

Given this example input file::

    6
    1
    8
    20
    11
    65
    2

.. -> input

Your program should generate this output file::

    1
    2
    6
    8
    11
    20
    65

.. -> output

    >>> input_lines = input.splitlines()
    >>> correct = '\n'.join(map(str, sorted(map(int, input_lines)))) + '\n'
    >>> output == correct
    True

This uses the syntax implemented in manuel.capture to capture a block of text into a variable (the one named after “->”).

Whenever a line of the structure ”.. -> VAR” is detected, the text of the previous block will be stored in the given variable.

Of course, lines that start with ”.. ” are reST comments, so when the document is rendered with docutils or Sphinx, the tests will dissapear and only the intended document contents will remain. Like so:

Challenge 1
===========

Write a program that sorts the numbers in a file.


Example
-------

Given this example input file::

    6
    1
    8
    20
    11
    65
    2

Your program should generate this output file::

    1
    2
    6
    8
    11
    20
    65

Code Blocks

Sphinx and other docutils extensions provide a “code-block” directive, which allows inlined snippets of code in reST documents.

The manuel.codeblock module provides the ability to execute the contents of Python code-blocks. For example:

.. code-block:: python

    print('hello')

If the code-block generates some sort of error...

.. code-block:: python

    print(does_not_exist)

...that error will be reported:

>>> document.process_with(m, globs={})
Traceback (most recent call last):
    ...
NameError: name 'does_not_exist' is not defined

If you find that you want to include a code-block in a document but don’t want Manuel to execute it, use manuel.ignore to ignore that particular block.

Docutils Code Blocks

Sphinx and docutils have different ideas of how code blocks should be spelled. Manuel supports the docutils-style code blocks too.

.. code:: python

    a = 1

Docutils options after the opening of the code block are also allowed:

.. code:: python
    :class: hidden

    a = 1

Invisible Code Blocks

At times you’ll want to have a block of code that is executed but not displayed in the rendered document (like some setup for later examples).

When using doctest’s native format (“>>>”) that’s easy to do, you just put the code in a reST comment, like so:

.. this is some setup, it is hidden in a reST comment

    >>> a = 5
    >>> b = a + 3

However, if you want to include a relatively large chunk of Python, you’d rather use a code-block, but that means that it will be included in the rendered document. Instead, manuel.codeblock also understands a variant of the code-block directive that is actually a reST comment: ”.. invisible-code-block:: python”:

.. invisible-code-block:: python

    a = 5
    b = a + 3

Note

The “invisible-code-block” directive will work with either one or two colons. The reason is that reST processers (like docutils and Sphinx) will generate an error for unrecognized directives (like invisible-code-block). Therefore you can use a single colon and the line will be interpreted as a comment instead.

Footnotes

The manuel.footnote module provides an implementation of reST footnote handling, but instead of just plain text, the footnotes can contain any syntax Manuel can interpret including doctests.

>>> import manuel.footnote
>>> m = manuel.footnote.Manuel()

Here’s an example of combining footnotes with doctests:

Here we reference a footnote. [1]_

    >>> x
    42

Here we reference another. [2]_

    >>> x
    100

.. [1] This is a test footnote definition.

    >>> x = 42

.. [2] This is another test footnote definition.

    >>> x = 100

.. [3] This is a footnote that will never be executed.

    >>> raise RuntimeError('nooooo!')

It is also possible to reference more than one footnote on a single line.

This line has several footnotes on it. [1]_ [2]_ [3]_

    >>> z
    105

A little prose to separate the examples.

.. [1] Do something

    >>> w = 3

.. [2] Do something

    >>> x = 5

.. [3] Do something

    >>> y = 7

    >>> z = w * x * y

Ignoring Blocks

Occasionally the need arises to ignore a block of markup that would otherwise be parsed by a Manuel plug-in.

For example, this document has a code-block that will generate a syntax error:

The following is invalid Python.

.. code-block:: python

    def foo:
        pass

We can see that when executed, the SyntaxError escapes.

>>> import manuel.codeblock
>>> m = manuel.codeblock.Manuel()
>>> document.process_with(m, globs={})
  File "<memory>:4", line 2
     def foo:
            ^
SyntaxError: invalid syntax

The manuel.ignore module provides a way to ignore parts of a document using a directive ”.. ignore-next-block”.

Because Manuel plug-ins are executed in the order they are accumulated, we want manuel.ignore to be the base Manuel object, with any additional plug-ins added to it.

import manuel.ignore
import manuel.doctest
m = manuel.ignore.Manuel()
m += manuel.codeblock.Manuel()
m += manuel.doctest.Manuel()

If we add an ignore marker to the block we don’t want processed...

The following is invalid Python.

.. ignore-next-block
.. code-block:: python

    def foo:
        pass

...the error goes away.

>>> document.process_with(m, globs={})
>>> print(document.formatted())

Ignoring Literal Blocks

Ignoring literal blocks is a little more involved:

Here is some invalid Python:

.. ignore-next-block

::

   >>> lambda: x=1

Test Isolation

One of the advantages of unittest over doctest is that the individual tests are isolated from one-another.

In large doctests (like this one) you may want to keep later tests from depending on incidental details of earlier tests, preventing the tests from becoming brittle and harder to change.

Test isolation is one approach to reducing this intra-doctest coupling. The manuel.isolation module provides a plug-in to help.

The ”.. reset-globs” directive resets the globals in the test:

We define a variable.

    >>> x = 'hello'

It is still defined.

    >>> print(x)
    hello

Now we can reset the globals...

.. reset-globs

...and the name binding will be gone:

    >>> print(x)
    Traceback (most recent call last):
        ...
    NameError: name 'x' is not defined

We can see that after the globals have been reset, the second “print(x)” line raises an error.

Of course, resetting to an empty set of global variables isn’t always what’s wanted. In that case there is a ”.. capture-globs” directive that saves a baseline set of globals that will be restored at each reset.

We define a variable.

    >>> x = 'hello'

It is still defined.

    >>> print(x)
    hello

We can capture the currently defined globals:

.. capture-globs

Of course capturing the globals doesn't disturb them.

    >>> print(x)
    hello

Now if we define a new global...

    >>> y = 'goodbye'
    >>> print(y)
    goodbye

.. reset-globs

...it will disappear after a reset.

    >>> print(y)
    Traceback (most recent call last):
        ...
    NameError: name 'y' is not defined

But the captured globals will still be defined.

    >>> print(x)
    hello

Identifying Test Cases

If you want parts of a document to be individually accessible as test cases (to be able to run just a particular subset of them, for example), a parser can create a region that marks the beginning of a new test case.

Two ways of identifying test cases are included in manuel.testcase:

  1. by section headings
  2. by explicit ”.. test-case: NAME” markers.

Grouping Tests by Heading

First Section
=============

Some prose.

    >>> print('first test case')

Some more prose.

    >>> print('still in the first test case')

Second Section
==============

Even more prose.

    >>> print('second test case')

Given the above document, if you’re using zope.testing’s testrunner (located in bin/test), you could run just the tests in the second section with this command:

bin/test -t "file-name.txt:Second Section"

Or, exploiting the fact that -t does a regex search (as opposed to a match):

bin/test -t file-name.txt:Second

Grouping Tests Explicitly

If you would like to identify test cases separately from sections, you can identify them with a marker:

First Section
=============

The following test will be in a test case that is not individually
identifiable.

    >>> print('first test case (unidentified)')

Some more prose.

.. test-case: first-named-test-case

    >>> print('first identified test case')


Second Section
==============

The test case markers don't have to immediately proceed a test.

.. test-case: second-named-test-case

Even more prose.

    >>> print('second identified test case')

Again, given the above document and zope.testing, you could run just the second set of tests with this command:

bin/test -t file-name.txt:second-named-test-case

Or, exploiting the fact that -t does a regex search again:

bin/test -t file-name.txt:second

Even though the tests are individually accessable doesn’t mean that they can’t all be run at the same time:

bin/test -t file-name.txt

Also, if you create a hierarchy of names, you can run groups of tests at a time. For example, lets say that you append “-important” to all your really important tests, you could then run the important tests for a single document like so:

bin/test -t 'file-name.txt:.*-important$'

or all the “important” tests no matter what file they are in:

bin/test -t '-important$'

Both Methods

You can also combine more than one test case identification method if you want. Here’s an example of building a Manuel stack that has doctests and both flavors of test case identification:

import manuel.doctest
import manuel.testcase

m = manuel.doctest.Manuel()
m += manuel.testcase.SectionManuel()
m += manuel.testcase.MarkerManuel()