Manuel lets you mix and match traditional doctests with custom test syntax.
Several plug-ins are included that provide new test syntax (see Included Functionality). You can also create your own plug-ins.
For example, if you’ve ever wanted to include a large chunk of Python in a doctest but were irritated by all the “>>>” and ”...” prompts required, you’d like the manuel.codeblock module. It lets you execute code using Sphinx-style ”.. code-block:: python” directives. The markup looks like this:
.. code-block:: python
import foo
def my_func(bar):
return foo.baz(bar)
Incidentally, the implementation of manuel.codeblock is only 23 lines of code.
The plug-ins included in Manuel make good examples while being quite useful in their own right. The Manuel documentation makes extensive use of them as well. Follow the “Show Source” link to the left to see the reST source of this document.
For a large example of creating test syntax, take a look at the FIT Table Example or for all the details, Theory of Operation.
To see how to get Manuel wired up see Getting Started.
Contents
Manuel includes several plug-ins out of the box:
The plug-ins used for a test are composed together using the “+” operator. Let’s say you wanted a test that used doctest syntax as well as footnotes. You would create a Manuel instance to use like this:
import manuel.doctest
import manuel.footnote
m = manuel.doctest.Manuel()
m += manuel.footnote.Manuel()
You would then pass the Manuel instance to a manuel.testing.TestSuite, including the names of documents you want to process:
manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')
The simplest way to get started with Manuel is to use unittest to run your tests:
import manuel.codeblock
import manuel.doctest
import manuel.testing
import unittest
def test_suite():
m = manuel.doctest.Manuel()
m += manuel.codeblock.Manuel()
return manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')
if __name__ == '__main__':
unittest.TextTestRunner().run(test_suite())
If you want to use a more featureful test runner you can use zope.testing’s test runner (usable stand-alone – it isn’t dependent on the Zope application server). Create a file named tests.py with a test_suite() function that returns a test suite.
The suite can be either a manuel.testing.TestSuite object or a unittest.TestSuite as demonstrated below.
import manuel.codeblock
import manuel.doctest
import manuel.testing
def test_suite():
suite = unittest.TestSuite()
# here you add your other tests to the suite...
# now you can add the Manuel tests
m = manuel.doctest.Manuel()
m += manuel.codeblock.Manuel()
suite.addTest(manuel.testing.TestSuite(m,
'test-one.txt', 'test-two.txt'))
return suite
If you know out how to make Manuel work with other test runners (nose, py.test, etc.), please send me an email and I’ll expand this section.
Manuel has its own manuel.testing.TestClass class that manuel.testing.TestSuite uses. If you want to customize it, you can pass in your own class to TestSuite.
import os.path
import manuel.testing
class StripDirsTestCase(manuel.testing.TestCase):
def shortDescription(self):
return os.path.basename(str(self))
suite = manuel.testing.TestSuite(
m, path_to_test, TestCase=StripDirsTestCase)
>>> list(suite)[0].shortDescription()
'bugs.txt'
Manuel is all about making testable documents and well-documented tests. Of course, Python’s doctest module is a long-standing fixture in that space, so it only makes sense for Manuel to support doctest syntax.
Handling doctests is easy:
import manuel.doctest
m = manuel.doctest.Manuel()
suite = manuel.testing.TestSuite(m, 'my-doctest.txt')
Of course you can mix in other Manuel syntax plug-ins as well (including ones you write yourself).
import manuel.doctest
import manuel.codeblock
m = manuel.doctest.Manuel()
m += manuel.codeblock.Manuel()
suite = manuel.testing.TestSuite(m, 'my-doctest-with-code-blocks.txt')
The manuel.doctest.Manuel constructor also takes optionflags and checker arguments.
m = manuel.doctest.Manuel(optionflags=optionflags, checker=checker)
See the doctest documentation for more information about the available options and output checkers
Note
zope.testing.renormalizing provides an OutputChecker for smoothing out differences between actual and expected output for things that are hard to control (like memory addresses and time). See the module’s doctests for more information on how it works. Here’s a short example that smoothes over the differences between CPython’s and PyPy’s NameError messages:
import re
import zope.testing.renormalizing
checker = zope.testing.renormalizing.RENormalizing([
(re.compile(r"NameError: global name '([a-zA-Z0-9_]+)' is not defined"),
r"NameError: name '\1' is not defined"),
])
When writing documentation the need often arises to describe the contents of files or other non-Python information. You may also want to put that information under test. manuel.capture helps with that.
For example, if you were writing the problems for a programming contest, you might want to describe the input and output files for each challenge, but you want to be sure that your examples are correct.
To do that you might write your document like this:
Challenge 1
===========
Write a program that sorts the numbers in a file.
Example
-------
Given this example input file::
6
1
8
20
11
65
2
.. -> input
Your program should generate this output file::
1
2
6
8
11
20
65
.. -> output
>>> input_lines = input.splitlines()
>>> correct = '\n'.join(map(str, sorted(map(int, input_lines)))) + '\n'
>>> output == correct
True
This uses the syntax implemented in manuel.capture to capture a block of text into a variable (the one named after “->”).
Whenever a line of the structure ”.. -> VAR” is detected, the text of the previous block will be stored in the given variable.
Of course, lines that start with ”.. ” are reST comments, so when the document is rendered with docutils or Sphinx, the tests will dissapear and only the intended document contents will remain. Like so:
Challenge 1
===========
Write a program that sorts the numbers in a file.
Example
-------
Given this example input file::
6
1
8
20
11
65
2
Your program should generate this output file::
1
2
6
8
11
20
65
Sphinx and other docutils extensions provide a “code-block” directive, which allows inlined snippets of code in reST documents.
The manuel.codeblock module provides the ability to execute the contents of Python code-blocks. For example:
.. code-block:: python
print('hello')
If the code-block generates some sort of error...
.. code-block:: python
print(does_not_exist)
...that error will be reported:
>>> document.process_with(m, globs={})
Traceback (most recent call last):
...
NameError: name 'does_not_exist' is not defined
If you find that you want to include a code-block in a document but don’t want Manuel to execute it, use manuel.ignore to ignore that particular block.
Sphinx and docutils have different ideas of how code blocks should be spelled. Manuel supports the docutils-style code blocks too.
.. code:: python
a = 1
Docutils options after the opening of the code block are also allowed:
.. code:: python
:class: hidden
a = 1
At times you’ll want to have a block of code that is executed but not displayed in the rendered document (like some setup for later examples).
When using doctest’s native format (“>>>”) that’s easy to do, you just put the code in a reST comment, like so:
.. this is some setup, it is hidden in a reST comment
>>> a = 5
>>> b = a + 3
However, if you want to include a relatively large chunk of Python, you’d rather use a code-block, but that means that it will be included in the rendered document. Instead, manuel.codeblock also understands a variant of the code-block directive that is actually a reST comment: ”.. invisible-code-block:: python”:
.. invisible-code-block:: python
a = 5
b = a + 3
Note
The “invisible-code-block” directive will work with either one or two colons. The reason is that reST processers (like docutils and Sphinx) will generate an error for unrecognized directives (like invisible-code-block). Therefore you can use a single colon and the line will be interpreted as a comment instead.
The manuel.footnote module provides an implementation of reST footnote handling, but instead of just plain text, the footnotes can contain any syntax Manuel can interpret including doctests.
>>> import manuel.footnote
>>> m = manuel.footnote.Manuel()
Here’s an example of combining footnotes with doctests:
Here we reference a footnote. [1]_
>>> x
42
Here we reference another. [2]_
>>> x
100
.. [1] This is a test footnote definition.
>>> x = 42
.. [2] This is another test footnote definition.
>>> x = 100
.. [3] This is a footnote that will never be executed.
>>> raise RuntimeError('nooooo!')
It is also possible to reference more than one footnote on a single line.
This line has several footnotes on it. [1]_ [2]_ [3]_
>>> z
105
A little prose to separate the examples.
.. [1] Do something
>>> w = 3
.. [2] Do something
>>> x = 5
.. [3] Do something
>>> y = 7
>>> z = w * x * y
Occasionally the need arises to ignore a block of markup that would otherwise be parsed by a Manuel plug-in.
For example, this document has a code-block that will generate a syntax error:
The following is invalid Python.
.. code-block:: python
def foo:
pass
We can see that when executed, the SyntaxError escapes.
>>> import manuel.codeblock
>>> m = manuel.codeblock.Manuel()
>>> document.process_with(m, globs={})
File "<memory>:4", line 2
def foo:
^
SyntaxError: invalid syntax
The manuel.ignore module provides a way to ignore parts of a document using a directive ”.. ignore-next-block”.
Because Manuel plug-ins are executed in the order they are accumulated, we want manuel.ignore to be the base Manuel object, with any additional plug-ins added to it.
import manuel.ignore
import manuel.doctest
m = manuel.ignore.Manuel()
m += manuel.codeblock.Manuel()
m += manuel.doctest.Manuel()
If we add an ignore marker to the block we don’t want processed...
The following is invalid Python.
.. ignore-next-block
.. code-block:: python
def foo:
pass
...the error goes away.
>>> document.process_with(m, globs={})
>>> print(document.formatted())
Ignoring literal blocks is a little more involved:
Here is some invalid Python:
.. ignore-next-block
::
>>> lambda: x=1
One of the advantages of unittest over doctest is that the individual tests are isolated from one-another.
In large doctests (like this one) you may want to keep later tests from depending on incidental details of earlier tests, preventing the tests from becoming brittle and harder to change.
Test isolation is one approach to reducing this intra-doctest coupling. The manuel.isolation module provides a plug-in to help.
The ”.. reset-globs” directive resets the globals in the test:
We define a variable.
>>> x = 'hello'
It is still defined.
>>> print(x)
hello
Now we can reset the globals...
.. reset-globs
...and the name binding will be gone:
>>> print(x)
Traceback (most recent call last):
...
NameError: name 'x' is not defined
We can see that after the globals have been reset, the second “print(x)” line raises an error.
Of course, resetting to an empty set of global variables isn’t always what’s wanted. In that case there is a ”.. capture-globs” directive that saves a baseline set of globals that will be restored at each reset.
We define a variable.
>>> x = 'hello'
It is still defined.
>>> print(x)
hello
We can capture the currently defined globals:
.. capture-globs
Of course capturing the globals doesn't disturb them.
>>> print(x)
hello
Now if we define a new global...
>>> y = 'goodbye'
>>> print(y)
goodbye
.. reset-globs
...it will disappear after a reset.
>>> print(y)
Traceback (most recent call last):
...
NameError: name 'y' is not defined
But the captured globals will still be defined.
>>> print(x)
hello
If you want parts of a document to be individually accessible as test cases (to be able to run just a particular subset of them, for example), a parser can create a region that marks the beginning of a new test case.
Two ways of identifying test cases are included in manuel.testcase:
First Section
=============
Some prose.
>>> print('first test case')
Some more prose.
>>> print('still in the first test case')
Second Section
==============
Even more prose.
>>> print('second test case')
Given the above document, if you’re using zope.testing’s testrunner (located in bin/test), you could run just the tests in the second section with this command:
bin/test -t "file-name.txt:Second Section"
Or, exploiting the fact that -t does a regex search (as opposed to a match):
bin/test -t file-name.txt:Second
If you would like to identify test cases separately from sections, you can identify them with a marker:
First Section
=============
The following test will be in a test case that is not individually
identifiable.
>>> print('first test case (unidentified)')
Some more prose.
.. test-case: first-named-test-case
>>> print('first identified test case')
Second Section
==============
The test case markers don't have to immediately proceed a test.
.. test-case: second-named-test-case
Even more prose.
>>> print('second identified test case')
Again, given the above document and zope.testing, you could run just the second set of tests with this command:
bin/test -t file-name.txt:second-named-test-case
Or, exploiting the fact that -t does a regex search again:
bin/test -t file-name.txt:second
Even though the tests are individually accessable doesn’t mean that they can’t all be run at the same time:
bin/test -t file-name.txt
Also, if you create a hierarchy of names, you can run groups of tests at a time. For example, lets say that you append “-important” to all your really important tests, you could then run the important tests for a single document like so:
bin/test -t 'file-name.txt:.*-important$'
or all the “important” tests no matter what file they are in:
bin/test -t '-important$'
You can also combine more than one test case identification method if you want. Here’s an example of building a Manuel stack that has doctests and both flavors of test case identification:
import manuel.doctest
import manuel.testcase
m = manuel.doctest.Manuel()
m += manuel.testcase.SectionManuel()
m += manuel.testcase.MarkerManuel()