Plugins

Contexts features an experimental ‘plugin’ interface for user-customisable behaviour.

The plugin interface

For documentation purposes, the full plugin interface is defined in PluginInterface. Currently, plugins are required to implement the initialise() method (to determine whether the plugin is active in the current test run). The rest of the plugin interface is optional.

Contexts’s plugin support is implemented as an ordered list of plugin classes. Each time a plugin hook is called, each plugin is called in turn. Plugins which do not implement the hook are skipped. The first return value is used as the return value of the aggregated plugin calls - when a plugin returns a value from a hook, all the remaining plugins in the list are skipped. This means that a given plugin is able to override the behaviour of plugins which follow it in the list.

The plugin lifecycle

Because plugins can override one another, the ordering of the list matters. The interface defines a classmethod entitled locate(), which you can implement to insert your plugin before or after another plugin.

Plugins are given control over whether or not they appear in the list. All plugins must define an initialise(args, environ) method, and return either True or False to signal whether they want to appear in the list. You may also define a setup_parser(parser) method to modify the argparse.ArgumentParser instance that is used to parse command-line arguments.

Very occasionally, it is necessary for plugins to modify the behaviour of other plugin objects (see FailuresOnly for an example) - you can define a request_plugins() generator method to request the current instances of some other plugin classes from the test runner.

Progress notifications

There exist a number of plugin hooks which are called when progress through the test run reaches certain points. These methods include test_run_started(), context_ended(), assertion_failed(), and so on.

These hooks are typically used to report progress to the user. It’s not recommended to return a value from these methods, unless you want to prevent other plugins from being told about the progress.

Identifying test objects

When the test runner sees a file, folder, class, or method, it queries the list of plugins to find out whether it should run it, using the identify_folder(), identify_file(), identify_class(), and identify_method() hooks. The expected return values from these methods are defined as constants in the contexts.plugin_interface module: TEST_FILE, CONTEXT, ASSERTION and so on.

After all the modules have been imported, the process_module_list() hook is called, which plugins can use to inject their own test modules, or remove modules that should not be run. There are also process_class_list() and process_assertion_list() hooks.

Other hooks

There are a few extra plugin hooks to override the way modules are imported (see AssertionRewritingImporter for an example) and to set the exit code for the process.

Registering a plugin

Once you’ve written your plugin, you can register it with Contexts using the contexts.plugins Setuptools entry point:

from setuptools import setup

setup(
    # ...
    entry_points = {
        'contexts.plugins': ['MyPluginClass = my_package.my_module:MyPluginClass']
    }
    # ...
)

The plugin API

class contexts.plugin_interface.PluginInterface

Defines the interface for plugins.

You do not need to inherit from this class in your own plugins. Just create a new class and implement initialise (and one or more other hooks).

classmethod locate()

Called before the plugin is instantiated, to determine where it should appear in the list of plugins. The ordering of this list matters. If a plugin returns a (non-None) value from a given method, plugins later in the list will not get called.

Plugins may return a 2-tuple of (left, right). Here, left is a plugin class which this plugin wishes to follow, and right is a class the plugin wishes to precede. Either or both of the values may be None, to indicate that the plugin does not mind what it comes before or after, respectively. Returning None from this method is equivalent to returning (None, None).

setup_parser(parser)

Called before command-line arguments are parsed.

Parameters:parser – An instance of argparse.ArgumentParser. Plugins may mutate parser in order to set it up to expect the options the plugin needs to configure itself. See the standard library documentation for argparse for more information.
initialise(args, environ)

Called after command-line arguments are parsed.

Parameters:
Returns:

A boolean. Returning True will cause the plugin to be added to the list of plugins for this test run. Returning False will prevent this.
request_plugins()

Called after all plugins have been initialised.

Plugins which need to modify the behaviour of other plugins may request instances of those plugins from the framework.

This must be a generator method. Yield an iterable of other plugin classes, and you will be sent a dictionary mapping those classes to the active instances of those plugins. Requested plugins that do not have an active instance will not be present in the dict.

test_run_started()

Called at the beginning of a test run.

test_run_ended()

Called at the end of a test run.

suite_started(module)

Called at the start of a test module.

Parameters:module – The Python module (an instance of ModuleType) that is about to be run.
suite_ended(module)

Called at the end of a test module.

Parameters:module – The Python module (an instance of ModuleType) that was just run.
test_class_started(cls)

Called when a test class begins its run.

A test class may contain one or more test contexts. (Test classes with examples will generally contain more than one.)

Parameters:cls – The class object that is being run.
test_class_ended(cls)

Called when a test class ends its run.

Parameters:cls – The class object that is being run.
test_class_errored(cls, exception)

Called when a test class unexpectedly errors.

Parameters:
  • cls – The class object that is being run.
  • exception – The exception that got caused the error.
context_started(cls, example)

Called when a test context begins its run.

Parameters:
  • cls – The class object of the test being run.
  • example – The current example, which may be NO_EXAMPLE if it is not a parametrised test.
context_ended(cls, example)

Called when a test context completes its run.

Parameters:
  • cls – The class object of the test being run.
  • example – The current example, which may be NO_EXAMPLE if it is not a parametrised test.
context_errored(cls, example, exception)

Called when a test context (not an assertion) throws an exception.

Parameters:
  • cls – The class object of the test being run.
  • example – The current example, which may be NO_EXAMPLE if it is not a parametrised test.
  • exception – The exception that caused the error.
assertion_started(func)

Called when an assertion begins.

Parameters:func – The assertion method being run.
assertion_passed(func)

Called when an assertion passes.

Parameters:func – The assertion method being run.
assertion_errored(func, exception)

Called when an assertion throws an exception.

Parameters:
  • func – The assertion method being run.
  • exception – The exception that caused the error.
assertion_failed(func, exception)

Called when an assertion throws an AssertionError.

Parameters:
  • func – The assertion method being run.
  • exception – The exception that caused the failure.
unexpected_error(exception)

Called when an error occurs outside of a Context or Assertion.

Parameters:exception – The exception that caused the failure.
get_object_to_run()

Called before the start of the test run, when the test runner wants to know what it should run.

This method should return one of:
  • a class - the test runner will run the identified methods in this class.
  • a file path as a string - the test runner will run the identified classes in this file.
  • a folder path as a string - the test runner will run the identified files and subfolders in this folder.
  • None - the plugin doesn’t want to choose what to run.
identify_folder(folder)

Called when the test runner encounters a folder and wants to know if it should run the tests in that folder.

Parameters:str (folder) – The full path of the folder which the test runner wants to be identified
This method should return one of:
  • TEST_FOLDER - plugin wishes the folder to be treated as a test folder
  • None - plugin does not wish to identify the folder (though other plugins may still cause it to be run)
identify_file(file)

Called when the test runner encounters a file and wants to know if it should run the tests in that file.

Parameters:str (file) – The full path of the file which the test runner wants to be identified.
This method should return one of:
  • TEST_FILE - plugin wishes the file to be imported and run as a test file
  • None - plugin does not wish to identify the file (though other plugins may still cause it to be run)
identify_class(cls)

Called when the test runner encounters a class and wants to know if it should treat it as a test class.

Parameters:cls – The class object which the test runner wants to be identified.
This method should return one of:
  • CONTEXT - plugin wishes the class to be treated as a test class
  • None - plugin does not wish to identify the class (though other plugins may still cause it to be run)
identify_method(func)

Called when the test runner encounters a method on a test class and wants to know if it should run the method.

When a test class has a superclass, all the superclass’s methods will be passed in first.

Parameters:func – The unbound method (or bound classmethod) which the test runner wants to be identified
This method should return one of:
  • EXAMPLES - plugin wishes the method to be treated as an ‘examples’ method
  • SETUP - plugin wishes the method to be treated as an ‘establish’ method
  • ACTION - plugin wishes the method to be treated as a ‘because’
  • ASSERTION - plugin wishes the method to be treated as an assertion method
  • TEARDOWN - plugin wishes the method to be treated as a teardown method
  • None - plugin does not wish to identify the method (though other plugins may still cause it to be run)
process_module_list(modules)

A hook to change (or examine) the list of modules which will be run with the full list of found modules. Plugins may modify the list in-place by adding or removing modules.

Parameters:modules – A list of types.ModuleType.
process_class_list(module, classes)

A hook to change (or examine) the list of classes found in a module. Plugins may modify the list in-place by adding or removing classes.

Parameters:
  • module – The Python module in which the classes were found (an instance of types.ModuleType).
  • classes – A list of classes found in that module.
process_assertion_list(cls, functions)

A hook to change (or examine) the list of (unbound) assertion methods found in a class. Plugins may modify the list in-place by adding or removing functions.

Parameters:
  • cls – The test class in which the methods were found
  • functions – A list of unbound assertion methods found in that class
import_module(location, name)

Called when the test runner needs to import a module.

Arguments:
location: string. Path to the folder containing the module or package. name: string. Full name of the module, including dot-separated package names.
This method should return one of:
  • an imported module (an instance of types.ModuleType).

    This may be a reference to an existing module, or a “fake” generated module.

  • None, if the plugin is not able to import the module.

get_exit_code()

Called at the end of the test runner to obtain the exit code for the process.

This method should return one of:
  • An integer
  • None, if you do not want to override the default behaviour.
contexts.plugin_interface.TEST_FOLDER

Returned by plugins to indicate that a folder contains tests.

contexts.plugin_interface.TEST_FILE

Returned by plugins to indicate that a file contains tests.

contexts.plugin_interface.CONTEXT

Returned by plugins to indicate that a class is a test class.

contexts.plugin_interface.EXAMPLES

Returned by plugins to indicate that a method is an Examples method.

contexts.plugin_interface.SETUP

Returned by plugins to indicate that a method is a setup method.

contexts.plugin_interface.ACTION

Returned by plugins to indicate that a method is an action method.

contexts.plugin_interface.ASSERTION

Returned by plugins to indicate that a method is an assertion method.

contexts.plugin_interface.TEARDOWN

Returned by plugins to indicate that a method is a teardown method.

contexts.plugin_interface.NO_EXAMPLE

Passed to plugins when a class is not a parametrised test.