======
Manuel
======

Manuel lets you mix and match traditional doctests with custom test syntax.

Several plug-ins are included that provide new test syntax (see
:ref:`functionality`).  You can also create your own plug-ins.

For example, if you've ever wanted to include a large chunk of Python in a
doctest but were irritated by all the ">>>" and "..." prompts required, you'd
like the :mod:`manuel.codeblock` module.  It lets you execute code using
Sphinx-style "..  code-block:: python" directives.  The markup looks like
this::

    .. code-block:: python

        import foo

        def my_func(bar):
            return foo.baz(bar)

Incidentally, the implementation of :mod:`manuel.codeblock` is only 23 lines of
code.

The plug-ins included in Manuel make good examples while being quite useful in
their own right.  The Manuel documentation makes extensive use of them as well.
Follow the "Show Source" link to the left to see the `reST
<http://docutils.sourceforge.net/rst.html>`_ source of this document.

For a large example of creating test syntax, take a look at the
:ref:`fit-table-example` or for all the details, :ref:`theory-of-operation`.

To see how to get Manuel wired up see :ref:`getting-started`.

.. contents::


.. reset-globs
.. _functionality:

Included Functionality
======================

Manuel includes several plug-ins out of the box:

:ref:`manuel.capture <capture>`
    stores regions of a document in variables for later processing

:ref:`manuel.codeblock <code-blocks>`
    executes code in ".. code-block:: python" blocks

:ref:`manuel.doctest <doctest>`
    provides traditional doctest processing as a Manuel plug-in

:ref:`manuel.footnote <footnotes>`
    executes code in reST-style footnodes each time they're referenced (good
    for getting incidental code out of the main flow of a document)

:ref:`manuel.ignore <ignore>`
    ignores parts of a document while running tests

:ref:`manuel.isolation <isolation>`
    makes it easier to have test isolation in doctests

:ref:`manuel.testcase <testcase>`
    identify parts of tests as individual test cases so they can be run
    independently


.. reset-globs
.. _getting-started:

Getting Started
===============

The plug-ins used for a test are composed together using the "+" operator.
Let's say you wanted a test that used doctest syntax as well as footnotes.  You
would create a Manuel instance to use like this:

.. code-block:: python

    import manuel.doctest
    import manuel.footnote

    m = manuel.doctest.Manuel()
    m += manuel.footnote.Manuel()

You would then pass the Manuel instance to a :class:`manuel.testing.TestSuite`,
including the names of documents you want to process:

.. ignore-next-block
.. code-block:: python

    manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')


Using unittest
--------------

The simplest way to get started with Manuel is to use :mod:`unittest` to run
your tests:

.. code-block:: python

    import manuel.codeblock
    import manuel.doctest
    import manuel.testing
    import unittest

    def test_suite():
        m = manuel.doctest.Manuel()
        m += manuel.codeblock.Manuel()
        return manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')

    if __name__ == '__main__':
        unittest.TextTestRunner().run(test_suite())


Using zope.testing
------------------

If you want to use a more featureful test runner you can use zope.testing's
test runner (usable stand-alone -- it isn't dependent on the Zope application
server).  Create a file named :file:`tests.py` with a :func:`test_suite`
function that returns a test suite.

The suite can be either a :class:`manuel.testing.TestSuite` object or a
:class:`unittest.TestSuite` as demonstrated below.

.. code-block:: python

    import manuel.codeblock
    import manuel.doctest
    import manuel.testing

    def test_suite():
        suite = unittest.TestSuite()

        # here you add your other tests to the suite...

        # now you can add the Manuel tests
        m = manuel.doctest.Manuel()
        m += manuel.codeblock.Manuel()
        suite.addTest(manuel.testing.TestSuite(m,
            'test-one.txt', 'test-two.txt'))

        return suite


Others
------

If you know out how to make Manuel work with other test runners (nose, py.test,
etc.), please `send me an email`_ and I'll expand this section.

.. _send me an email: benji+manuel@benjiyork.com


Customizing the TestCase class
------------------------------

Manuel has its own :class:`manuel.testing.TestClass` class that
:class:`manuel.testing.TestSuite` uses. If you want to customize it, you
can pass in your own class to `TestSuite`.

.. code-block:: python

     import os.path
     import manuel.testing

     class StripDirsTestCase(manuel.testing.TestCase):
         def shortDescription(self):
             return os.path.basename(str(self))
     suite = manuel.testing.TestSuite(
         m, path_to_test, TestCase=StripDirsTestCase)

    >>> list(suite)[0].shortDescription()
    'bugs.txt'


.. reset-globs
.. _doctest:

Doctests
========

Manuel is all about making testable documents and well-documented tests.  Of
course, Python's doctest module is a long-standing fixture in that space, so it
only makes sense for Manuel to support doctest syntax.

Handling doctests is easy:

.. ignore-next-block
.. code-block:: python

    import manuel.doctest

    m = manuel.doctest.Manuel()
    suite = manuel.testing.TestSuite(m, 'my-doctest.txt')

Of course you can mix in other Manuel syntax plug-ins as well (including ones
you write yourself).

.. ignore-next-block
.. code-block:: python

    import manuel.doctest
    import manuel.codeblock

    m = manuel.doctest.Manuel()
    m += manuel.codeblock.Manuel()
    suite = manuel.testing.TestSuite(m, 'my-doctest-with-code-blocks.txt')

The :class:`manuel.doctest.Manuel` constructor also takes :data:`optionflags`
and :data:`checker` arguments.

.. ignore-next-block
.. code-block:: python

    m = manuel.doctest.Manuel(optionflags=optionflags, checker=checker)

See the `doctest documentation <http://docs.python.org/library/doctest.html>`_
for more information about the `available options
<http://docs.python.org/library/doctest.html#doctest-options>`_  and `output
checkers <http://docs.python.org/library/doctest.html#outputchecker-objects>`_


.. note::

   :mod:`zope.testing.renormalizing` provides an :class:`OutputChecker`
   for smoothing out differences between actual and expected output for things
   that are hard to control (like memory addresses and time).  See the
   `module's doctests <http://svn.zope.org/zope.testing/trunk/src/zope/testing/renormalizing.py?view=markup>`_
   for more information on how it works.  Here's a short example that
   smoothes over the differences between CPython's and PyPy's NameError
   messages:

.. code-block:: python

    import re
    import zope.testing.renormalizing
    checker = zope.testing.renormalizing.RENormalizing([
        (re.compile(r"NameError: global name '([a-zA-Z0-9_]+)' is not defined"),
         r"NameError: name '\1' is not defined"),
    ])


.. reset-globs
.. _capture:

Capturing Blocks
================

When writing documentation the need often arises to describe the contents of
files or other non-Python information.  You may also want to put that
information under test.  :mod:`manuel.capture` helps with that.

For example, if you were writing the problems for a programming contest, you
might want to describe the input and output files for each challenge, but you
want to be sure that your examples are correct.

To do that you might write your document like this:

::

    Challenge 1
    ===========

    Write a program that sorts the numbers in a file.


    Example
    -------

    Given this example input file::

        6
        1
        8
        20
        11
        65
        2

    .. -> input

    Your program should generate this output file::

        1
        2
        6
        8
        11
        20
        65

    .. -> output

        >>> input_lines = input.splitlines()
        >>> correct = '\n'.join(map(str, sorted(map(int, input_lines)))) + '\n'
        >>> output == correct
        True

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> import manuel.capture
    >>> m = manuel.capture.Manuel()
    >>> import manuel.doctest
    >>> m += manuel.doctest.Manuel()
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

This uses the syntax implemented in :mod:`manuel.capture` to capture a block of
text into a variable (the one named after "->").

Whenever a line of the structure ".. -> VAR" is detected, the text of the
*previous* block will be stored in the given variable.

.. the paragraph below could be phrased better

Of course, lines that start with ".. " are reST comments, so when the document
is rendered with docutils or Sphinx, the tests will dissapear and only the
intended document contents will remain.  Like so::

    Challenge 1
    ===========

    Write a program that sorts the numbers in a file.


    Example
    -------

    Given this example input file::

        6
        1
        8
        20
        11
        65
        2

    Your program should generate this output file::

        1
        2
        6
        8
        11
        20
        65


.. reset-globs
.. _code-blocks:

Code Blocks
===========

`Sphinx <http://sphinx.pocoo.org/>`_ and other docutils `extensions
<http://docutils.sourceforge.net/sandbox/code-block-directive/docs/syntax-highlight.html>`_
provide a `"code-block" directive <http://sphinx.pocoo.org/markup/code.html>`_,
which allows inlined snippets of code in reST documents.

The :mod:`manuel.codeblock` module provides the ability to execute the contents
of Python code-blocks.  For example::

    .. code-block:: python

        print('hello')

.. Let's create a reST document with a code block.

    >>> import manuel.codeblock
    >>> document = manuel.Document("""
    ... Here is a code-block:
    ...
    ... .. code-block:: python
    ...
    ...     x = 'hello'
    ...
    ... A little prose to separate the examples.
    ...
    ...     >>> print(x)
    ...     hello
    ...
    ... """)

.. Since the above document mixes code-blocks and doctests, we'll mix in the
   doctest handler.

    >>> import manuel.doctest
    >>> m = manuel.codeblock.Manuel()
    >>> m += manuel.doctest.Manuel()
    >>> document.process_with(m, globs={})

    Both code blocks were found (for a total of five regions -- text, block,
    text, block, and text):

    >>> len(list(document))
    5

    We can see that none of the tests in the document failed:

    >>> print(document.formatted())

If the code-block generates some sort of error...

.. code-block:: python

    .. code-block:: python

        print(does_not_exist)

.. -> source

    >>> document = manuel.Document(source, location='fake.txt')

.. the document above was specially formulated to have nothing before or after
   the code-block

    >>> document.source.startswith('.. code-block')
    True
    >>> document.source.endswith('print(does_not_exist)\n')
    True

...that error will be reported:

    >>> document.process_with(m, globs={})
    Traceback (most recent call last):
        ...
    NameError: name 'does_not_exist' is not defined

If you find that you want to include a code-block in a document but don't want
Manuel to execute it, use :ref:`manuel.ignore <ignore>` to ignore that
particular block.


.. reset-globs

Docutils Code Blocks
--------------------

Sphinx and docutils have different ideas of how code blocks should be spelled.
Manuel supports the docutils-style code blocks too.

::

    .. code:: python

        a = 1

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> import manuel.codeblock
    >>> m = manuel.codeblock.Manuel()
    >>> document.parse_with(m)
    >>> for region in document:
    ...     print((region.lineno, region.parsed or region.source))
    (1, <manuel.codeblock.CodeBlock object...>)

Docutils options after the opening of the code block are also allowed::

    .. code:: python
        :class: hidden

        a = 1

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> import manuel.codeblock
    >>> m = manuel.codeblock.Manuel()
    >>> document.parse_with(m)
    >>> for region in document:
    ...     print((region.lineno, region.parsed, region.parsed.source))
    (1, <manuel.codeblock.CodeBlock object...>, '\na = 1\n')


Invisible Code Blocks
---------------------

At times you'll want to have a block of code that is executed but not displayed
in the rendered document (like some setup for later examples).

When using doctest's native format (">>>") that's easy to do, you just put the
code in a reST comment, like so:

::

    .. this is some setup, it is hidden in a reST comment

        >>> a = 5
        >>> b = a + 3

However, if you want to include a relatively large chunk of Python, you'd
rather use a code-block, but that means that it will be included in the
rendered document.  Instead, :mod:`manuel.codeblock` also understands a variant
of the code-block directive that is actually a reST comment: "..
invisible-code-block:: python"::

    .. invisible-code-block:: python

        a = 5
        b = a + 3

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

.. note:: The "invisible-code-block" directive will work with either one or two
   colons.  The reason is that reST processers (like docutils and Sphinx) will
   generate an error for unrecognized directives (like invisible-code-block).
   Therefore you can use a single colon and the line will be interpreted as a
   comment instead.

.. the single-colon variant works too

    >>> document = manuel.Document("""
    ...
    ... .. invisible-code-block: python
    ...
    ...     raise RuntimeError('it worked!')
    ...
    ... """)
    >>> document.process_with(m, globs={})
    Traceback (most recent call last):
        ...
    RuntimeError: it worked!


.. reset-globs
.. _footnotes:

Footnotes
=========

The :mod:`manuel.footnote` module provides an implementation of reST footnote
handling, but instead of just plain text, the footnotes can contain any syntax
Manuel can interpret including doctests.

    >>> import manuel.footnote
    >>> m = manuel.footnote.Manuel()

Here's an example of combining footnotes with doctests:

.. so we also need the doctest Manuel plug-in

    >>> import manuel.doctest
    >>> m += manuel.doctest.Manuel()

::

    Here we reference a footnote. [1]_

        >>> x
        42

    Here we reference another. [2]_

        >>> x
        100

    .. [1] This is a test footnote definition.

        >>> x = 42

    .. [2] This is another test footnote definition.

        >>> x = 100

    .. [3] This is a footnote that will never be executed.

        >>> raise RuntimeError('nooooo!')

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

.. The order of examples in footnotes is preserved.  If not, the document below
   would generate an error because "a" won't be defined when "b = a + 1" is
   evaluated.

    >>> document = manuel.Document("""
    ... Here we want some imports to be done. [foo]_
    ...
    ...     >>> a + b
    ...     3
    ...
    ... A little prose to separate the examples.
    ...
    ... .. [foo] Do something
    ...
    ...     >>> a = 1
    ...
    ...     >>> b = a + 1
    ...
    ... """)
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

It is also possible to reference more than one footnote on a single line.

::

    This line has several footnotes on it. [1]_ [2]_ [3]_

        >>> z
        105

    A little prose to separate the examples.

    .. [1] Do something

        >>> w = 3

    .. [2] Do something

        >>> x = 5

    .. [3] Do something

        >>> y = 7

        >>> z = w * x * y

.. -> source2

    >>> document = manuel.Document(source)
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())


.. reset-globs
.. _ignore:

Ignoring Blocks
===============

.. reset-globs

Occasionally the need arises to ignore a block of markup that would otherwise
be parsed by a Manuel plug-in.

For example, this document has a code-block that will generate a syntax error::

    The following is invalid Python.

    .. code-block:: python

        def foo:
            pass

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> import manuel.codeblock
    >>> m = manuel.codeblock.Manuel()

We can see that when executed, the SyntaxError escapes.

    >>> import manuel.codeblock
    >>> m = manuel.codeblock.Manuel()
    >>> document.process_with(m, globs={})
    Traceback (most recent call last):
      ...
      File "<memory>:4", line 2
         def foo:
                ^
    SyntaxError: invalid syntax

The :mod:`manuel.ignore` module provides a way to ignore parts of a document
using a directive ".. ignore-next-block".

Because Manuel plug-ins are executed in the order they are accumulated, we want
:mod:`manuel.ignore` to be the base Manuel object, with any additional plug-ins
added to it.

.. code-block:: python

    import manuel.ignore
    import manuel.doctest
    m = manuel.ignore.Manuel()
    m += manuel.codeblock.Manuel()
    m += manuel.doctest.Manuel()

If we add an ignore marker to the block we don't want processed...

.. code-block:: python

    The following is invalid Python.

    .. ignore-next-block
    .. code-block:: python

        def foo:
            pass

.. -> source

    >>> document = manuel.Document(source)

...the error goes away.

    >>> document.process_with(m, globs={})
    >>> print(document.formatted())


Ignoring Literal Blocks
-----------------------

Ignoring literal blocks is a little more involved::

    Here is some invalid Python:

    .. ignore-next-block

    ::

       >>> lambda: x=1

.. -> source

    >>> document = manuel.Document(source)

    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

.. we want to be very sure that the above example without the ignore actually
   generates an error:

    >>> document = manuel.Document(document.source.replace(
    ...     '.. ignore-next-block', ''))
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())
    File "<memory>"...
    Exception raised:
          ...
         SyntaxError: ...


.. reset-globs
.. _isolation:

Test Isolation
==============

One of the advantages of unittest over doctest is that the individual tests are
isolated from one-another.

In large doctests (like this one) you may want to keep later tests from
depending on incidental details of earlier tests, preventing the tests from
becoming brittle and harder to change.

Test isolation is one approach to reducing this intra-doctest coupling.  The
:mod:`manuel.isolation` module provides a plug-in to help.

The ".. reset-globs" directive resets the globals in the test::

    We define a variable.

        >>> x = 'hello'

    It is still defined.

        >>> print(x)
        hello

    Now we can reset the globals...

    .. reset-globs

    ...and the name binding will be gone:

        >>> print(x)
        Traceback (most recent call last):
            ...
        NameError: name 'x' is not defined

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> import manuel.isolation
    >>> import manuel.doctest
    >>> m = manuel.isolation.Manuel()
    >>> m += manuel.doctest.Manuel(checker=checker)

We can see that after the globals have been reset, the second "print(x)" line
raises an error.

Of course, resetting to an empty set of global variables isn't always what's
wanted.  In that case there is a ".. capture-globs" directive that saves a
baseline set of globals that will be restored at each reset.

::

    We define a variable.

        >>> x = 'hello'

    It is still defined.

        >>> print(x)
        hello

    We can capture the currently defined globals:

    .. capture-globs

    Of course capturing the globals doesn't disturb them.

        >>> print(x)
        hello

    Now if we define a new global...

        >>> y = 'goodbye'
        >>> print(y)
        goodbye

    .. reset-globs

    ...it will disappear after a reset.

        >>> print(y)
        Traceback (most recent call last):
            ...
        NameError: name 'y' is not defined

    But the captured globals will still be defined.

        >>> print(x)
        hello

.. -> source

    >>> import manuel
    >>> document = manuel.Document(source)
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())


.. reset-globs
.. _testcase:

Identifying Test Cases
======================

If you want parts of a document to be individually accessible as test cases (to
be able to run just a particular subset of them, for example), a parser can
create a region that marks the beginning of a new test case.

Two ways of identifying test cases are included in :mod:`manuel.testcase`:

1. by section headings

2. by explicit ".. test-case: NAME" markers.

Grouping Tests by Heading
-------------------------

::

    First Section
    =============

    Some prose.

        >>> print('first test case')

    Some more prose.

        >>> print('still in the first test case')

    Second Section
    ==============

    Even more prose.

        >>> print('second test case')

.. -> source

    >>> import manuel
    >>> import manuel.testcase
    >>> document = manuel.Document(source)
    >>> m = manuel.testcase.SectionManuel()
    >>> m += manuel.doctest.Manuel()
    >>> document.process_with(m, globs={})
    >>> print(document.formatted())
    File "<memory>"...
    Failed example:
        print('first test case')
    Expected nothing
    Got:
        first test case
    File "<memory>"...
    Failed example:
        print('still in the first test case')
    Expected nothing
    Got:
        still in the first test case
    File "<memory>"...
    Failed example:
        print('second test case')
    Expected nothing
    Got:
        second test case

.. now lets see if the regions are grouped as we expect

    >>> import manuel.testing
    >>> for regions in manuel.testing.group_regions_by_test_case(document):
    ...     print((regions.location, regions.id))
    ('<memory>', None)
    ('<memory>', 'First Section')
    ('<memory>', 'Second Section')

Given the above document, if you're using zope.testing's testrunner (located in bin/test), you could run just the tests in the second section with this command::

    bin/test -t "file-name.txt:Second Section"

Or, exploiting the fact that -t does a regex search (as opposed to a match)::

    bin/test -t file-name.txt:Second


Grouping Tests Explicitly
-------------------------

If you would like to identify test cases separately from sections, you can
identify them with a marker::

    First Section
    =============

    The following test will be in a test case that is not individually
    identifiable.

        >>> print('first test case (unidentified)')

    Some more prose.

    .. test-case: first-named-test-case

        >>> print('first identified test case')


    Second Section
    ==============

    The test case markers don't have to immediately proceed a test.

    .. test-case: second-named-test-case

    Even more prose.

        >>> print('second identified test case')

.. -> source

    >>> document = manuel.Document(source)
    >>> m = manuel.testcase.MarkerManuel()
    >>> m += manuel.doctest.Manuel()
    >>> document.parse_with(m)
    >>> import six
    >>> for regions in manuel.testing.group_regions_by_test_case(document):
    ...     six.print_(regions.location, regions.id)
    <memory> None
    <memory> first-named-test-case
    <memory> second-named-test-case

Again, given the above document and zope.testing, you could run just the second
set of tests with this command::

    bin/test -t file-name.txt:second-named-test-case

Or, exploiting the fact that -t does a regex search again::

    bin/test -t file-name.txt:second

Even though the tests are individually accessable doesn't mean that they can't
all be run at the same time::

    bin/test -t file-name.txt

Also, if you create a hierarchy of names, you can run groups of tests at a
time.  For example, lets say that you append "-important" to all your really
important tests, you could then run the important tests for a single document
like so::

    bin/test -t 'file-name.txt:.*-important$'

or all the "important" tests no matter what file they are in::

    bin/test -t '-important$'

Both Methods
------------

You can also combine more than one test case identification method if you want.
Here's an example of building a Manuel stack that has doctests and both flavors
of test case identification:

.. code-block:: python

    import manuel.doctest
    import manuel.testcase

    m = manuel.doctest.Manuel()
    m += manuel.testcase.SectionManuel()
    m += manuel.testcase.MarkerManuel()

.. make sure above finds all the test cases appropriately

    >>> document.parse_with(m)
    >>> import six
    >>> for regions in manuel.testing.group_regions_by_test_case(document):
    ...     six.print_(regions.location, regions.id)
    <memory> None
    <memory> First Section
    <memory> first-named-test-case
    <memory> Second Section
    <memory> second-named-test-case


Further Reading
===============

.. toctree::
    :maxdepth: 1

    README.txt
    table-example.txt
    bugs.txt