Note
This Testing Taxonomy is a work in progress. The canonical version is currently at:
Contents
The testing tools taxonomy is a collection of links to Python testing libraries and tools. The testing tools taxonomy is maintained by Grig Gheorghiu and Michael Foord.
If you have any suggestions, corrections or additions for the taxonomy, please email them to the Testing in Python Mailing List.
This documented is written in ReStructured Text. The source is available on bitbucket if you want to contribute patches: Taxonomy project
| Tool | Category | Author | Part of | Claim to fame |
|---|---|---|---|---|
| unittest | unit testing | Steve Purcell | Python Standard Library | The main standard library testing framework and one of the oldest Python testing frameworks. Easy to use by people familiar with the xUnit frameworks; strong support for test organization and reuse via test suites. |
| unittest2 | unit testing | Michael Foord | A standalone port of the new features in unittest in Python 2.7 & 3.2 to work with Python 2.4-2.6. This includes automatic test discovery, enhanced command line features, shared fixtures (class and module level tearDown and setUp) and so on. | |
| doctest | unit testing | Tim Peters | Python Standard Library | Copy and paste output from shell session; Agile Documentation. Unit tests themselves can serve as documentation. Also see Django's approach. |
| py.test | unit testing | Holger Krekel | py.lib | No API! Automatic collection of tests; simple asserts; strong support for test fixture/state management; strong debugging support. |
| nose | unittest extensions | Jason Pellerin | Widely used testing framework built on top of unittest. nose provides an alternate test discovery and running process for unittest, one that is intended to mimic the behavior of py.test as much as is reasonably possible without resorting to too much magic. Friendly with unittest.TestCase-based tests and there are many plugins available. | |
| TestOOB | unittest extensions | Ori Peleg | unittest enhancements; test filtering via regex/glob patterns; reporting in XML/HTML; colorized output; runs pdb on failing tests; run in parallel in threads/processes; verbose asserts; report failures immediately; and a little more. | |
| Trial | unittest extensions | Jonathan Lange | Twisted | Extension of unittest to support writing asynchronous unit tests using Deferreds and new result types ('skip' and 'todo'). Includes a command-line program that does test discovery and integrates with doctest and coverage. |
| subunit | unittest extensions | Robert Collins | Transparently adds support for running unittest test cases/suites in a separate process : prevents system wide changes by a test destabilising the test runner. It also allows reporting from tests in another process into the unittest framework, giving a single integrated test environment. | |
| testresources | unittest extensions | Robert Collins | Provides a mechanism for managing 'resources' - expensive bits of infrastructure - that are needed by multiple tests. Resources are constructed and free on demand, but with an optional TestSuite?, the test run order is optimised to reduce the number of resource constructions and releases needed. Compatible with unittest. | |
| testtools | unittest extensions | Jonathan Lange | Useful extensions to unittest derived from custom extensions by projects such as Twisted and Bazaar. | |
| pyUnitPerf | performance testing | Grig Gheorghiu | Port of JUnitPerf. Transparently adds performance testing capabilities to existing unitttest-based test suites. | |
| Sancho | unit testing | MEMS | MEMS Exchange Tools | Sancho runs tests, and provides output for tests that fail. Sancho does not count tests passed or failed; targets projects that do not maintain failing tests. |
| Peckcheck | specification based unit testing | Darius Bacon | Like the unittest module, except that test methods may take arguments; you declare each argument with a default value which names a test-data generator and the peckcheck module will peck at your test methods with a bunch of generated values. | |
| testosterone | unit testing | Chad Whitacre | A manly test runner; command-line and curses(3) interfaces; the iPod of test runners. (More seriously, makes it easier to browse and inspect your test failures). | |
| zope.testing | unit testing | Zope3 Community | Powerful test runner that includes support for post-mortem debugging of test failures. Also includes profiling and coverage reporting. This is a standalone package that has no dependencies on Zope and works just fine with projects that don't use Zope. | |
| pry | unit testing | Nullcube | Elegant unit testing framework with built-in coverage analysis, profiling, micro-benchmarking and a powerful command-line interface. | |
| Pythoscope | unit testing | Michal Kwiatkowski | Tool that will automatically, or semi-automatically, generate unit tests for legacy systems written in Python. | |
| dutest | unittest extensions | Olemis Lang | An object oriented interface to retrieve unittest test cases out of doctests. Hides initialization from doctests by allowing setUp and tearDown for each interactive example. Allows control over all the options provided by doctest. Specialized classes allow selective test discovery across a package hierarchy. |
| Tool | Category | Author | Claim to fame |
|---|---|---|---|
| mock | mock testing, patching | Michael Foord | Library for simple mocking, based on "action -> assertion" rather than "record -> replay" used by other mocking libraries. mock includes support for temporarily replacing classes and other objects in modules under test with mock objects (monkey patching). |
| Ludibrio | mocks, stubs, spy, and dummies | Gustavo Rezende | Elegant test doubles framework in Python (mocks, stubs, spy, and dummies) |
| Python Mock | mock testing | Dave Kirby | Python Mock enables the easy creation of mock objects that can be used to emulate the behaviour of any class that the code under test depends on. You can set up expectations about the calls that are made to the mock object, and examine the history of calls made. This makes it easier to unit test classes in isolation. |
| PyMock | mock testing | Jeff Younker | PyMock is based on the Java easymock package. It uses a recording and replay model rather than using a specification language. Easymock lives up to its name compared to other mocking packages. PyMock takes advantage of python's dynamic nature for futher improvements. |
| minimock | mock testing, doctest | Ian Bicking | Embeds mock testing constructs inside doctest tests. |
| svnmock | mock testing | Collin Winter | Enables easier testing of Python programs that make use of Subversion's Python bindings. |
| Mocker | mock testing | Gustavo Niemeyer | Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies). Well-documented and fairly feature-complete. |
| Stubble | stub testing | Iwan Vosloo | Stubble allows you to write arbitrary classes for use as stubs instead of read classes while testing. Stubble lets you link a stub class loosely to the real class which it is a stub for. This information is then used to ensure that tests will break if there is a discrepancy between the interface supported by your stub class and that of the real class it stands in for. |
| Mox | mock testing | smiddlek, dglasser | Mox is based on EasyMock, a Java mock object framework. Mox will make mock objects for you, so you don't have to create your own! It mocks the public/protected interfaces of Python objects. You set up your mock objects expected behavior using a domain specific language (DSL), which makes it easy to use, understand, and refactor! |
| MockTest | mock testing | Tim Cuthbertson (gfxmonl | Mocktest allows you to mock / stub objects and make expectations about the methods you expect to be called as well as the arguments they should be called with. Expectations are very readable and expressive, and checked automatically. Any stubbed methods are reverted after each test case. Still under development, so subject to change. |
| Fudge | mock and stub testing | Kumar McMillan | A module for using fake objects (mocks, stubs, etc) to test real ones. Uses a declrative syntax like jMock whereby you set up expectations for how an object should be used. An error will raise if an expectation is not met. |
| Mockito for Python | mock and stub testing | Szczepan Faber | A port of the Mockito mocking framework to Python. (Technically speaking, Mockito is a Test Spy framework.) |
According to Wikipedia, "fuzz testing" (or "fuzzing") is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior.
| Tool | Author | Claim to fame |
|---|---|---|
| fusil | Victor Stinner | Fusil the fuzzer is a Python library used to write fuzzing programs. It helps to start process with a prepared environment (limit memory, environment variables, redirect stdout, etc.), start network client or server, and create mangled files. Fusil has many probes to detect program crash: watch process exit code, watch process stdout and syslog for text patterns (eg. "segmentation fault"), watch session duration, watch cpu usage (process and system load), etc. |
| Pester | Ivan Moore | Tests your tests by mutating source code and finding tests that don't fail! |
| Peach Fuzzer Framework | Michael Eddington | Peach can fuzz just about anything from .NET, COM/ActiveX, SQL, shared libraries/DLL's, network applications, web, you name it. |
| antiparser | dmckinney | The purpose of antiparser is to provide an API that can be used to model network protocols and file formats by their composite data types. Once a model has been created, the antiparser has various methods for creating random sets of data that deviates in ways that will ideally trigger software bugs or security vulnerabilities. |
| Taof (The Art Of Fuzzing) | rodrigomarcos | Taof is a GUI cross-platform Python generic network protocol fuzzer. It has been designed for minimizing set-up time during fuzzing sessions and it is specially useful for fast testing of proprietary or undocumented protocols. |
First, let's define some categories of Web testing tools:
| Tool | Category | Author | Part of | Claim to fame |
|---|---|---|---|---|
| Twill | Browser simulation & In-process | Titus Brown | Offers simple commands for navigating Web pages, posting forms and asserting conditions; can be used as shell script or Python module; can be used for unit-testing your own Web app; stress-test functionality; port of PBP; uses John J. Lee's mechanize. | |
Note
Although not a Python-specific testing tool, Selenium is nevertheless an important addition to any tester's arsenal. It relates to Python in that it can use a Twisted-based server to run tests against Web applications. Read a tutorial about it, followed by a quick update. With make_selenium.py script you can write Selenium tests in Python.