Proboscis is a Python test framework that extends Python’s built-in unittest module and Nose with features from TestNG.
Proboscis uses decorators similar to TestNG’s annotations instead of naming conventions to mark classes and functions as tests. If a class is decorated with proboscis.test, a single instance of the class is created and used to run all decorated methods, similar to TestNG. However, if the decorated class extends unittest.TestCase it is run using the traditional rules.
Like TestNG, Proboscis allows tests to be added to groups so they can be organized and run independently of the code layout (similar to tags in Nose). It also lets tests cleanly and explicitly declare dependencies on other tests, opening the door for functional and integration testing (if you are maintaining Python tests which use funny names like “010_start”, “020_connect”, etc. or find that everything breaks when you try to move a module to a different package, you’re running tests like this now and need to be using a tool that supports them). Of course, Proboscis works fine for unit testing as well.
Proboscis also supports factory methods, which operate similar to those in TestNG.
Proboscis will use Nose instead of the unittest module if it is available; otherwise, it uses only the core Python libraries so that it can also run on Iron Python and Jython. Some Nose plugins work with Proboscis out of the box, while others may take some prodding or not work at all. Proboscis works only in Python 2, but Python 3 support is pending.
Proboscis runs imported test functions or classes decorated with the proboscis.test decorator. Decorated classes extending unittest.TestCase run exactly like they do in Nose / unittest.
This means traditional Python unit test classes can run as-is in Proboscis provided they are decorated.
For example:
import unittest
from proboscis.asserts import assert_equal
from proboscis import test
import utils
@test(groups=["unit", "numbers"])
class TestIsNegative(unittest.TestCase):
"""Confirm that utils.is_negative works correctly."""
def test_should_return_true_for_negative_numbers(self):
self.assertTrue(utils.is_negative(-47))
def test_should_return_false_for_positive_numbers(self):
self.assertFalse(utils.is_negative(56))
def test_should_return_false_for_zero(self):
self.assertFalse(utils.is_negative(0))
You can also attach the proboscis.test decorator to functions to run them by themselves:
@test(groups=["unit", "strings"])
def test_reverse():
"""Make sure our complex string reversal logic works."""
original = "hello"
expected = "olleh"
actual = utils.reverse(original)
assert_equal(expected, actual)
Unlike Nose Proboscis requires all tests modules must be imported directly in code, so using it requires you write a start-up script like the following:
def run_tests():
from proboscis import TestProgram
from tests import unit
# Run Proboscis and exit.
TestProgram().run_and_exit()
if __name__ == '__main__':
run_tests()
Assuming this is named something like “run_test.py” you can run it like so:
$ python run_tests.py
test_should_return_false_for_positive_numbers (examples.unit.tests.unit.TestIsNegative) ... ok
test_should_return_false_for_zero (examples.unit.tests.unit.TestIsNegative) ... ok
test_should_return_true_for_negative_numbers (examples.unit.tests.unit.TestIsNegative) ... ok
Make sure our complex string reversal logic works. ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
TestProgram.run_and_exit() expects to be used in scripts like this and takes command line arguments into account (Note: it’s called “run_and_exit()” because to run the tests it calls Nose which then calls unittest, which calls sys.exit() on completion and forces the program to exit).
Normally, all tests are run, but we can use the “–group” command line parameter to run only a certain group (and the groups it depends on) instead:
$ python run_tests.py --group=strings
Make sure our complex string reversal logic works. ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
If you want to run multiple specific groups, use the “–group” parameter more than once.
You can also use the “–show-plan” argument to get a preview of how Proboscis will run the tests:
$ python run_tests.py --show-plan
* * * Test Plan * * *
<class 'examples.unit.tests.unit.TestIsNegative'>
Confirm that utils.is_negative works correctly.
groups = [unit,numbers]
enabled = True
depends_on_groups = []
depends_on = set([])
runs_after = set([])
<function test_reverse at 0x110e032a8>
Make sure our complex string reversal logic works.
groups = [unit,strings]
enabled = True
depends_on_groups = []
depends_on = set([])
runs_after = set([])
Unused arguments get passed along to Nose or the unittest module, which means its possible to run some plugins designed for them. However, Proboscis is by nature invasive and intentionally and unintentionally breaks certain features of Nose and unittest (such as test discovery) so your mileage may vary.
Proboscis is more useful for higher level tests which may have dependencies on each other or need to run in a guaranteed order.
Nose can order tests lexically but the effect is difficult to maintain, especially when working with multiple modules. Additionally, if one test performs some sort of initialization to produce a state required by other tests and fails, the dependent tests run despite having no chance of succeeding. These additional failures pollute the results making the true problem harder to see.
In Proboscis, if one tests depends on another which fails, the dependent test raises Nose’s SkipTest or calls unittest’s skipTest() automatically, making it easier to track down the real problem. If neither feature is available (as is the case with Python 2.5), it simply raises an assertion with a message beginning with the word “SKIPPED.”
The following example shows how to write a test with dependencies to test a fictitious web service that stores user profiles. The service allows admin users to create and delete users and allows users to edit a profile picture.
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
test_user = None
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@test(groups=["service.initialization"])
def initialize_database_and_server():
"""Creates a local database and starts up the web service."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "user.initialization"],
depends_on_groups=["service.initialization"])
def create_user():
random.seed()
global test_user
test_user = None
new_user_config = generate_new_user_config()
admin = mymodule.get_admin_client()
test_user = admin.create_user(new_user_config)
assert_equal(test_user.username, new_user_config["username"])
assert_true(test_user.id is not None)
assert_true(isinstance(test_user.id, int))
@test(groups=["user", "user.tests"],
depends_on_groups=["user.initialization"])
def user_cant_connect_with_wrong_password():
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'fdgggdsds'})
@test(groups=["user", "user.tests"],
depends_on_groups=["service.initialization"])
class WhenConnectingAsANormalUser(unittest.TestCase):
def setUp(self):
self.client = mymodule.login({
'username':test_user.username, 'password':'password'})
def test_auth_create(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.create_user,
generate_new_user_config())
def test_auth_delete(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.delete_user,
test_user.id)
def test_change_profile_image(self):
"""Test changing a client's profile image."""
self.assertEquals("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
self.assertEquals("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"], depends_on_groups=["user.tests"],
always_run=True)
def delete_user():
"""Delete the user."""
test_user = None
if test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'password'})
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"],
depends_on_groups=["service.initialization", "service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
Our initialization code runs in three phases: first, we create the database, second, we start the web service (assuming its some kind of daemon we can run programmatically) and third we create a new user. The function “initialize_database_and_server” is in the group “service.initialization”, while the function “create_user” is in the group “user.initialization”. Note that the “create_user” depends on “initialize_database_and_server”, so Proboscis guarantees it runs after.
The meat of the test is where we run some operations against the user. These classes and functions are marked as depending on the “user.initialization” group and so run later.
The tests which clean everything up depend on the groups “user.tests” and “service.tests” respectively. We also set the “always_run” property to true so that if a test in the group they depend on fails they will still run. Since the “delete_user” test function could run even when the “create_user” test function fails to even make a user, we add some code to check the status of the global “test_user” object and skip it if it was never set.
When we run the run_test.py script, we see everything is ordered correctly:
$ python run_tests.py
Creates a local database and starts up the web service. ... ok
proboscis.case.FunctionTest (create_user) ... ok
proboscis.case.FunctionTest (user_cant_connect_with_wrong_password) ... ok
Make sure the given client cannot perform admin actions.. ... ok
Make sure the given client cannot perform admin actions.. ... ok
Test changing a client's profile image. ... ok
Delete the user. ... SKIP: User tests were never run.
Shut down the web service and destroys the database. ... ok
----------------------------------------------------------------------
Ran 8 tests in 0.001s
OK (SKIP=1)
In some frameworks initialization code is run as part of a “fixture”, or something else which is a bit different than a test, but in Proboscis our initialization code is a test itself and can be covered with assertions.
Let’s say there’s an error and the web service starts up. In a traditional testing framework, you’d see a stream of error messages as every test failed. In Proboscis, you get this:
$ python run_tests.py
Creates a local database and starts up the web service. ... ERROR
proboscis.case.FunctionTest (create_user) ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
proboscis.case.FunctionTest (user_cant_connect_with_wrong_password) ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Make sure the given client cannot perform admin actions.. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Make sure the given client cannot perform admin actions.. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Test changing a client's profile image. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Delete the user. ... SKIP: User tests were never run.
Shut down the web service and destroys the database. ... ok
======================================================================
ERROR: Creates a local database and starts up the web service.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/tim.simpson/Work/github/TimSimpsonR/python-proboscis/.tox/py26/lib/python2.6/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/Users/tim.simpson/Work/github/TimSimpsonR/python-proboscis/.tox/py26/lib/python2.6/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "../../examples/example1/tests/service_tests.py", line 55, in initialize_database_and_server
mymodule.start_web_server()
File "../../examples/example1/mymodule.py", line 49, in bad_start_web_server
raise RuntimeError("Error starting service.")
RuntimeError: Error starting service.
----------------------------------------------------------------------
Ran 8 tests in 0.002s
FAILED (SKIP=6, errors=1)
The example above is pretty group heavy- in some cases, a group is created just to establish a single dependency.
Its also possible to establish dependencies without groups by listing a function or class directly as a dependencies. The code below runs identically to the example above but does so without groups:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
test_user = None
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "service.tests"],
depends_on=[initialize_web_server])
def create_user():
random.seed()
global test_user
test_user = None
new_user_config = generate_new_user_config()
admin = mymodule.get_admin_client()
test_user = admin.create_user(new_user_config)
assert_equal(test_user.username, new_user_config["username"])
assert_true(test_user.id is not None)
assert_true(isinstance(test_user.id, int))
@test(groups=["user", "user.tests", "service.tests"],
depends_on=[create_user])
def user_cant_connect_with_wrong_password():
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'fdgggdsds'})
@test(groups=["user", "user.tests", "service.tests"],
depends_on=[create_user])
class WhenConnectingAsANormalUser(unittest.TestCase):
def setUp(self):
self.client = mymodule.login({
'username':test_user.username, 'password':'password'})
def test_auth_create(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.create_user,
generate_new_user_config())
def test_auth_delete(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.delete_user,
test_user.id)
def test_change_profile_image(self):
"""Test changing a client's profile image."""
self.assertEquals("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
self.assertEquals("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"], depends_on_groups=["user.tests"],
always_run=True)
def delete_user():
if test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'password'})
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
The example above creates the test user as a global variable so it can pass it between the tests which use it. Because unittest creates a new instance of the class “WhenConnectingAsANormalUser” for each method it runs, we can’t run the code to create the user in the setUp method and store it in that class either.
An gross alternative would be to merge all of the tests which require a user into a single function, but this would understandably be a bit gross. It also would not be equivalent, since if one test failed, no other tests would get a chance to run (for example, if test represented by “test_auth_delete” unittest would output a single test failure, and the test for “change_profile_image” would never run). It would also be uncharitable to anyone who had to maintain the code.
There’s another way in Proboscis, though, which is to run test methods in the style of TestNG by putting the @test decorator on both the class and test methods and making sure the class does not extend unittest.TestCase.
When the TestNG method is used, a single instance of a class is created and used to run each method.
If we do this, we can combine all of the tests which require the user into one class as follows:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import after_class
from proboscis import before_class
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "service.tests"])
class NormalUserTests(object):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@before_class
def create_user(self):
"""Create a user."""
random.seed()
global test_user
test_user = None
new_user_config = self.generate_new_user_config()
admin = mymodule.get_admin_client()
self.test_user = admin.create_user(new_user_config)
assert_equal(self.test_user.username, new_user_config["username"])
assert_true(self.test_user.id is not None)
assert_true(isinstance(self.test_user.id, int))
@after_class(always_run=True)
def delete_user(self):
if self.test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(self.test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'password'})
@test
def cant_login_with_wrong_password(self):
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'blah'})
@test
def successful_login(self):
self.client = mymodule.login({
'username':self.test_user.username, 'password':'password'})
@test(depends_on=[successful_login])
def a_normal_user_cant_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.create_user,
self.generate_new_user_config())
@test(depends_on=[successful_login])
def a_normal_user_cant_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.delete_user,
self.test_user.id)
@test(depends_on=[successful_login])
def change_profile_image(self):
"""Test changing a client's profile image."""
assert_equal("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
assert_equal("spam.jpg", self.client.get_profile_image())
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
@before_class and @after_class work just like the @test decorator and accept the same arguments, but also tell the method to run either before and after all other methods in the given class.
If a test can fit into one class, its usually best to write it this way.
Consider what happens if we want to test the admin user- before, we would have had to duplicate our test code for the normal user or somehow gotten the same test code to run twice while we altered the global test_user variable in between.
However using the newly refactored code testing for the admin user can be accomplished fairly easy via subclassing:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import after_class
from proboscis import before_class
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
class UserTests(object):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@before_class
def create_user(self):
"""Create a user."""
random.seed()
global test_user
test_user = None
new_user_config = self.generate_new_user_config()
admin = mymodule.get_admin_client()
self.test_user = admin.create_user(new_user_config)
assert_equal(self.test_user.username, new_user_config["username"])
assert_true(self.test_user.id is not None)
assert_true(isinstance(self.test_user.id, int))
@after_class(always_run=True)
def delete_user(self):
if self.test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(self.test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'password'})
def cant_login_with_wrong_password(self):
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'blah'})
@test
def successful_login(self):
self.client = mymodule.login({
'username':self.test_user.username, 'password':'password'})
@test(depends_on=[successful_login])
def change_profile_image(self):
"""Test changing a client's profile image."""
assert_equal("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
assert_equal("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"])
class NormalUserTests(UserTests):
@test(depends_on=[UserTests.successful_login])
def a_normal_user_cant_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.create_user,
self.generate_new_user_config())
@test(depends_on=[UserTests.successful_login])
def a_normal_user_cant_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.delete_user,
self.test_user.id)
@test(groups=["user", "service.tests"])
class AdminUserTests(UserTests):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"admin"
}
return new_user_config
@test(depends_on=[UserTests.successful_login])
def an_admin_user_can_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
self.new_user = self.client.create_user(self.generate_new_user_config())
# Make sure it actually logs in.
self.new_user_client = mymodule.login({
'username':self.new_user.username, 'password':'password'})
@test(depends_on=[an_admin_user_can_create_users])
def an_admin_user_can_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
self.client.delete_user(self.new_user.id)
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
Its possible to create empty test entries that link groups together using the proboscis.register function without a class or function. A good place to do (as well as store other bits of configuration) is in the start up script you write for Proboscis. Here’s an example:
from proboscis import register
from proboscis import TestProgram
def run_tests():
from tests import service_tests
# Now create some groups of groups.
register(groups=["fast"], depends_on_groups=["unit"])
register(groups=["integration"],
depends_on_groups=["service.initialize",
"service.tests",
"service.shutdown"])
register(groups=["slow"],
depends_on_groups=["fast", "integration"])
# Run Proboscis and exit.
TestProgram().run_and_exit()
if __name__ == '__main__':
run_tests()
Here the groups “fast”, “integration”, and “slow” are created as simple dependencies on other groups. This makes it possible to, for example, run all “slow” tests with the following command:
python runtests.py --group=slow
Decorates a test class or function to cause Proboscis to run it.
The behavior differs depending the target:
If put on a stand-alone function, the function will run by itself.
If put on a class inheriting unittest.TestCase, then the class will run just like a normal unittest class by using the method names and instantiate a new instance of the class for each test method.
If the class does not inherit from unittest.TestCase, the class will be instantiated once and this instance will be passed to each method decorated with @test (this increases encapsulation over using class fields as the instance can not be accessed outside of its methods).
Note that due to how decorators work its impossible to know if a function is or is not part of a class; thus if a class method is decorated with test but its class is not then ProboscisTestMethodNotDecorated will be raised.
| Parameters: |
|
|---|
Like @test but indicates this should run before other class methods.
All of the arguments sent to @test work with this decorator as well.
Like @test but indicates this should run after other class methods.
All of the arguments sent to @test work with this decorator as well.
This will be skipped if a class method test fails; set always_run if that is not desired. See issue #5.
Bases: nose.core.TestProgram
Use this to run Proboscis.
Translates the Proboscis test registry into types used by Nose or unittest in order to run the program.
Most arguments to this are simply passed to Nose or unittest’s TestProgram class.
For most cases using the default arguments works fine.
| Parameters: |
|
|---|
Creates a suite runnable by unittest.
Given argv, records the “–group” options.
| Parameters: |
|
|---|
Returns a copy of param argv with the –group options removed. This is useful if argv needs to be passed to another program such as Nose.
Calls unittest or Nose to run all tests.
unittest will call sys.exit on completion.
Prints information on test entries and the order they will run.
Bases: nose.plugins.skip.SkipTest
Raise this to skip a test. If Nose is available its SkipTest is used. Otherwise Proboscis creates its own which class that calls unittest.TestCase.skipTest. If that method isn’t available (anything under 2.7) then skipping does not work and test errors are presented.
Registers a test in proboscis’s default registry.
| Parameters: | home – The target class or function. |
|---|
This also allows all of the parameters used by the @test decorator.
This function works differently than a decorator as it allows the class or function which is being registered to appear in the same call as all of the options.
Its designed to make it easier to register class or functions with Proboscis after they’re defined.
Decorates a function which returns new instances of Test classes.
Bases: object
Stores test information.
All of Proboscis’s decorators (@test, @before_class, etc) and the register function use a default instance of this class, however its also possible to instantiate multiple copies and add tests to them directly.
Adds the group to the registry if it does not exist.
| Parameters: | group_name – The group to create. |
|---|
Returns a TestGroup given its name.
| Parameters: | group_name – Group to return. |
|---|
Registers a bit of code (or nothing) to be run / ordered as a test.
Registering a test with nothing allows for the creation of groups of groups, which can be useful for organization.
When proboscis.register is called it chains to this method bound to the global default registry.
Turns a function into a Proboscis test instance factory.
A factory returns a list of test class instances. Proboscis runs all factories at start up and sorts the instances like normal tests.
| Parameters: | func – the function to be added. |
|---|
Wipes the registry.
Bases: exceptions.Exception
This denotes a very common error that seems somewhat unavoidable due to the fact it isn’t possible to know if a method is bound or not in Python when you decorate it.
Assert functions with a parameter order of actual_value, expected_value.
This module contains many stand-ins for functions in Nose.tools. It is also a clone of TestNG’s Assert class with the static methods changed to functions, and the term “equals” changed to simply “equal” to be more Pythonic.
There are also a few original assertions methods and the class Check.
This module should be preferred when Nose is not always available.
Asserts that the two values are equal.
| Parameters: |
|
|---|
Asserts that the given condition is false.
| Parameters: |
|
|---|
Asserts that the two variables share the same identity.
| Parameters: |
|
|---|
Asserts that the given value is None.
| Parameters: |
|
|---|
Asserts that the two variables has different identities.
| Parameters: |
|
|---|
Asserts that a value is anything other than None.
| Parameters: |
|
|---|
Asserts that the two values are not equal.
| Parameters: |
|
|---|
Asserts that the given value is True.
| Parameters: |
|
|---|
Calls function and fails the test if an exception is not raised.
Unlike nose.Tool’s assert_raises or TestCase.assertRaises the given exception type must match the exactly: if the raised exception is a subclass the test will fail. For example, it fails if the exception_type param is “Exception” but “RuntimeException” is raised. To be less demanding use assert_raises_instance.
| Parameters: |
|
|---|
Calls function and fails the test if an exception is not raised.
The exception thrown must only be an instance of the given type. This means if “Exception” is expected but “RuntimeException” is raised the test will still pass. For a stricter function see assert_raises.
| Parameters: |
|
|---|
Fails a test.
| Parameters: | message – The message to display. |
|---|
Unlike the other functions in this module the message argument is required.
Bases: object
Used to test and record multiple failing asserts in a single function.
Usually its best to write numerous small methods with a single assert in each one, but sometimes this ideal isn’t possible and multiple asserts must be made in the same function.
In some cases, such as when all properties of a returned object are being interrogated, these asserts do not depend on each other and having the test stop after the first one can be a bother if the task was run on a CI server or somewhere else.
This class solves that by saving any assert failures and raising one giant assert at the end of a with block.
To use it, write something like:
some_obj = ...
with Check() as check:
check.equal(some_obj.a, "A")
check.equal(some_obj.b, "B")
check.equal(some_obj.c, "C")
At the end of the with block if any checks failed one assertion will be raised containing inside it the stack traces for each assertion.
If instances are not used in a with block any failed assert will raise instantly.
Identical to assert_equal.
Identical to fail.
Identical to assert_false.
Identical to assert_is_none.
Identical to assert_is_not.
Identical to assert_is_not_none.
Identical to assert_is.
Identical to assert_not_equal.
Identical to assert_raises.
Identical to assert_raises_instance.
Identical to assert_true.