Unit and Documentation Testing

Unit Tests

Documentation Tests

As described in Showing Code Snippets, Sphinx integrates with Python’s doctest module to help ensure that both the documentation and underlying library are correct. Doctests consist of short snippets of code along with their expected output. A doctest passes if the actual output of the snippet matches its expected output. For instance, a doctest that 1 + 1 correctly produces 2 could be written as:

Below, we show an example of addition in practice:

>>> 1 + 1
2

Later, we will consider more advanced operators.

The blank lines above and below the doctest separate it from the surrounding text, while the expected output appears immediately below the relevant code.

To run the doctests in the QInfer documentation using Linux or OS X:

$ cd doc/
$ make doctest

To run the doctests on Windows using PowerShell, use .\make instead:

PS > cd doc/
PS > .\make doctest

As with the unit tests, doctests are automatically run on pull requests, to help ensure the correctness of contributed documentation.

Test Annotations

A doctest snippet may be annotated with one or more comments that change the behavior of that test. The doctest documentation goes into far more detail, but the two we will commonly need are # doctest: +SKIP and # doctest: +ELLIPSIS. The former causes a test to be skipped entirely. Skipping tests can be useful if the output of a doctest is random, for instance.

The second, +ELLIPSIS, causes any ellipsis (...) in the expected output to act as a wild card. For instance, both of the following doctests would pass:

>>> print([1, 2, 3]) 
[1, ..., 3]
>>> print([1, 2, 3, 4, 'foo', 3]) 
[1, ..., 3]

Doctest Annoyances

There are a few annoyances that come along with writing tests based on string-equivalence of outputs, in particular for a cross-platform and 2/3 compatible library. In particular:

• NumPy, 2to3 and type/class: Python 2 and 3 differ on whether the type of NumPy ndarray prints as type (Python 2) class (Python 3):

  >>> print(type(np.array([1])))
  <type 'numpy.ndarray'> # Python 2
  <class 'numpy.ndarray'> # Python 3
  

  Thus, to write a doctest that checks if something is a NumPy array or not, it is preferred to use an isinstance check instead:

  >>> isinstance(np.array([1]), np.ndarray)
  True
  

  Though this sacrifices some on readability, it gains on portability and correctness.

• int versus long in array shapes: The Windows and Linux versions of NumPy behave differently with respect to when a NumPy shape is represented as a tuple of int versus a tuple of long values. This can cause doctests to choke on spurious L suffixes:

  >>> print(np.zeros((1, 10)).shape)
  (1, 10) # tuple of int
  (1L, 10L) # tuple of long
  

  Since int and long respect ==, however, the same trick as above can help:

  >>> np.zeros((1, 10)).shape == (1, 10)
  True