Proboscis runs imported test functions or classes decorated with the proboscis.test decorator. Decorated classes extending unittest.TestCase run exactly like they do in Nose / unittest.
This means traditional Python unit test classes can run as-is in Proboscis provided they are decorated.
For example:
import unittest
from proboscis.asserts import assert_equal
from proboscis import test
import utils
@test(groups=["unit", "numbers"])
class TestIsNegative(unittest.TestCase):
"""Confirm that utils.is_negative works correctly."""
def test_should_return_true_for_negative_numbers(self):
self.assertTrue(utils.is_negative(-47))
def test_should_return_false_for_positive_numbers(self):
self.assertFalse(utils.is_negative(56))
def test_should_return_false_for_zero(self):
self.assertFalse(utils.is_negative(0))
You can also attach the proboscis.test decorator to functions to run them by themselves:
@test(groups=["unit", "strings"])
def test_reverse():
"""Make sure our complex string reversal logic works."""
original = "hello"
expected = "olleh"
actual = utils.reverse(original)
assert_equal(expected, actual)
Unlike Nose Proboscis requires all tests modules must be imported directly in code, so using it requires you write a start-up script like the following:
def run_tests():
from proboscis import TestProgram
from tests import unit
# Run Proboscis and exit.
TestProgram().run_and_exit()
if __name__ == '__main__':
run_tests()
Assuming this is named something like “run_test.py” you can run it like so:
$ python run_tests.py
test_should_return_false_for_positive_numbers (examples.unit.tests.unit.TestIsNegative) ... ok
test_should_return_false_for_zero (examples.unit.tests.unit.TestIsNegative) ... ok
test_should_return_true_for_negative_numbers (examples.unit.tests.unit.TestIsNegative) ... ok
Make sure our complex string reversal logic works. ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
TestProgram.run_and_exit() expects to be used in scripts like this and takes command line arguments into account (Note: it’s called “run_and_exit()” because to run the tests it calls Nose which then calls unittest, which calls sys.exit() on completion and forces the program to exit).
Normally, all tests are run, but we can use the “–group” command line parameter to run only a certain group (and the groups it depends on) instead:
$ python run_tests.py --group=strings
Make sure our complex string reversal logic works. ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
If you want to run multiple specific groups, use the “–group” parameter more than once.
You can also use the “–show-plan” argument to get a preview of how Proboscis will run the tests:
$ python run_tests.py --show-plan
* * * Test Plan * * *
<class 'examples.unit.tests.unit.TestIsNegative'>
Confirm that utils.is_negative works correctly.
groups = [unit,numbers]
enabled = True
depends_on_groups = []
depends_on = set([])
runs_after = set([])
<function test_reverse at 0x110e032a8>
Make sure our complex string reversal logic works.
groups = [unit,strings]
enabled = True
depends_on_groups = []
depends_on = set([])
runs_after = set([])
Unused arguments get passed along to Nose or the unittest module, which means its possible to run some plugins designed for them. However, Proboscis is by nature invasive and intentionally and unintentionally breaks certain features of Nose and unittest (such as test discovery) so your mileage may vary.
Proboscis is more useful for higher level tests which may have dependencies on each other or need to run in a guaranteed order.
Nose can order tests lexically but the effect is difficult to maintain, especially when working with multiple modules. Additionally, if one test performs some sort of initialization to produce a state required by other tests and fails, the dependent tests run despite having no chance of succeeding. These additional failures pollute the results making the true problem harder to see.
In Proboscis, if one tests depends on another which fails, the dependent test raises Nose’s SkipTest or calls unittest’s skipTest() automatically, making it easier to track down the real problem. If neither feature is available (as is the case with Python 2.5), it simply raises an assertion with a message beginning with the word “SKIPPED.”
The following example shows how to write a test with dependencies to test a fictitious web service that stores user profiles. The service allows admin users to create and delete users and allows users to edit a profile picture.
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
test_user = None
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@test(groups=["service.initialization"])
def initialize_database_and_server():
"""Creates a local database and starts up the web service."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "user.initialization"],
depends_on_groups=["service.initialization"])
def create_user():
random.seed()
global test_user
test_user = None
new_user_config = generate_new_user_config()
admin = mymodule.get_admin_client()
test_user = admin.create_user(new_user_config)
assert_equal(test_user.username, new_user_config["username"])
assert_true(test_user.id is not None)
assert_true(isinstance(test_user.id, int))
@test(groups=["user", "user.tests"],
depends_on_groups=["user.initialization"])
def user_cant_connect_with_wrong_password():
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'fdgggdsds'})
@test(groups=["user", "user.tests"],
depends_on_groups=["service.initialization"])
class WhenConnectingAsANormalUser(unittest.TestCase):
def setUp(self):
self.client = mymodule.login({
'username':test_user.username, 'password':'password'})
def test_auth_create(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.create_user,
generate_new_user_config())
def test_auth_delete(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.delete_user,
test_user.id)
def test_change_profile_image(self):
"""Test changing a client's profile image."""
self.assertEquals("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
self.assertEquals("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"], depends_on_groups=["user.tests"],
always_run=True)
def delete_user():
"""Delete the user."""
test_user = None
if test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'password'})
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"],
depends_on_groups=["service.initialization", "service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
Our initialization code runs in three phases: first, we create the database, second, we start the web service (assuming its some kind of daemon we can run programmatically) and third we create a new user. The function “initialize_database_and_server” is in the group “service.initialization”, while the function “create_user” is in the group “user.initialization”. Note that the “create_user” depends on “initialize_database_and_server”, so Proboscis guarantees it runs after.
The meat of the test is where we run some operations against the user. These classes and functions are marked as depending on the “user.initialization” group and so run later.
The tests which clean everything up depend on the groups “user.tests” and “service.tests” respectively. We also set the “always_run” property to true so that if a test in the group they depend on fails they will still run. Since the “delete_user” test function could run even when the “create_user” test function fails to even make a user, we add some code to check the status of the global “test_user” object and skip it if it was never set.
When we run the run_test.py script, we see everything is ordered correctly:
$ python run_tests.py
Creates a local database and starts up the web service. ... ok
proboscis.case.FunctionTest (create_user) ... ok
proboscis.case.FunctionTest (user_cant_connect_with_wrong_password) ... ok
Make sure the given client cannot perform admin actions.. ... ok
Make sure the given client cannot perform admin actions.. ... ok
Test changing a client's profile image. ... ok
Delete the user. ... SKIP: User tests were never run.
Shut down the web service and destroys the database. ... ok
----------------------------------------------------------------------
Ran 8 tests in 0.001s
OK (SKIP=1)
In some frameworks initialization code is run as part of a “fixture”, or something else which is a bit different than a test, but in Proboscis our initialization code is a test itself and can be covered with assertions.
Let’s say there’s an error and the web service starts up. In a traditional testing framework, you’d see a stream of error messages as every test failed. In Proboscis, you get this:
$ python run_tests.py
Creates a local database and starts up the web service. ... ERROR
proboscis.case.FunctionTest (create_user) ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
proboscis.case.FunctionTest (user_cant_connect_with_wrong_password) ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Make sure the given client cannot perform admin actions.. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Make sure the given client cannot perform admin actions.. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Test changing a client's profile image. ... SKIP: Failure in <function initialize_database_and_server at 0x110bbc140>
Delete the user. ... SKIP: User tests were never run.
Shut down the web service and destroys the database. ... ok
======================================================================
ERROR: Creates a local database and starts up the web service.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/tim.simpson/Work/github/TimSimpsonR/python-proboscis/.tox/py26/lib/python2.6/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/Users/tim.simpson/Work/github/TimSimpsonR/python-proboscis/.tox/py26/lib/python2.6/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "../../examples/example1/tests/service_tests.py", line 55, in initialize_database_and_server
mymodule.start_web_server()
File "../../examples/example1/mymodule.py", line 49, in bad_start_web_server
raise RuntimeError("Error starting service.")
RuntimeError: Error starting service.
----------------------------------------------------------------------
Ran 8 tests in 0.002s
FAILED (SKIP=6, errors=1)
The example above is pretty group heavy- in some cases, a group is created just to establish a single dependency.
Its also possible to establish dependencies without groups by listing a function or class directly as a dependencies. The code below runs identically to the example above but does so without groups:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
test_user = None
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "service.tests"],
depends_on=[initialize_web_server])
def create_user():
random.seed()
global test_user
test_user = None
new_user_config = generate_new_user_config()
admin = mymodule.get_admin_client()
test_user = admin.create_user(new_user_config)
assert_equal(test_user.username, new_user_config["username"])
assert_true(test_user.id is not None)
assert_true(isinstance(test_user.id, int))
@test(groups=["user", "user.tests", "service.tests"],
depends_on=[create_user])
def user_cant_connect_with_wrong_password():
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'fdgggdsds'})
@test(groups=["user", "user.tests", "service.tests"],
depends_on=[create_user])
class WhenConnectingAsANormalUser(unittest.TestCase):
def setUp(self):
self.client = mymodule.login({
'username':test_user.username, 'password':'password'})
def test_auth_create(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.create_user,
generate_new_user_config())
def test_auth_delete(self):
"""Make sure the given client cannot perform admin actions.."""
self.assertRaises(mymodule.AuthException, self.client.delete_user,
test_user.id)
def test_change_profile_image(self):
"""Test changing a client's profile image."""
self.assertEquals("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
self.assertEquals("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"], depends_on_groups=["user.tests"],
always_run=True)
def delete_user():
if test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':test_user.username, 'password':'password'})
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
The example above creates the test user as a global variable so it can pass it between the tests which use it. Because unittest creates a new instance of the class “WhenConnectingAsANormalUser” for each method it runs, we can’t run the code to create the user in the setUp method and store it in that class either.
An gross alternative would be to merge all of the tests which require a user into a single function, but this would understandably be a bit gross. It also would not be equivalent, since if one test failed, no other tests would get a chance to run (for example, if test represented by “test_auth_delete” unittest would output a single test failure, and the test for “change_profile_image” would never run). It would also be uncharitable to anyone who had to maintain the code.
There’s another way in Proboscis, though, which is to run test methods in the style of TestNG by putting the @test decorator on both the class and test methods and making sure the class does not extend unittest.TestCase.
When the TestNG method is used, a single instance of a class is created and used to run each method.
If we do this, we can combine all of the tests which require the user into one class as follows:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import after_class
from proboscis import before_class
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
@test(groups=["user", "service.tests"])
class NormalUserTests(object):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@before_class
def create_user(self):
"""Create a user."""
random.seed()
global test_user
test_user = None
new_user_config = self.generate_new_user_config()
admin = mymodule.get_admin_client()
self.test_user = admin.create_user(new_user_config)
assert_equal(self.test_user.username, new_user_config["username"])
assert_true(self.test_user.id is not None)
assert_true(isinstance(self.test_user.id, int))
@after_class(always_run=True)
def delete_user(self):
if self.test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(self.test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'password'})
@test
def cant_login_with_wrong_password(self):
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'blah'})
@test
def successful_login(self):
self.client = mymodule.login({
'username':self.test_user.username, 'password':'password'})
@test(depends_on=[successful_login])
def a_normal_user_cant_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.create_user,
self.generate_new_user_config())
@test(depends_on=[successful_login])
def a_normal_user_cant_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.delete_user,
self.test_user.id)
@test(depends_on=[successful_login])
def change_profile_image(self):
"""Test changing a client's profile image."""
assert_equal("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
assert_equal("spam.jpg", self.client.get_profile_image())
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
@before_class and @after_class work just like the @test decorator and accept the same arguments, but also tell the method to run either before and after all other methods in the given class.
If a test can fit into one class, its usually best to write it this way.
Consider what happens if we want to test the admin user- before, we would have had to duplicate our test code for the normal user or somehow gotten the same test code to run twice while we altered the global test_user variable in between.
However using the newly refactored code testing for the admin user can be accomplished fairly easy via subclassing:
"""
User service tests.
This is a test for a fictitious user web service which has rich client bindings
written in Python.
It assumes we have an existing test database which we can run the web service
against, using the function "mymodule.start_web_server()."
After spinning up the service, the test creates a new user and tests various
CRUD actions. Since its a test database, it is OK
to leave old users in the system but we try to always delete them if possible
at the end of the test.
"""
from datetime import datetime
import random
import types
import unittest
import mymodule
from proboscis.asserts import assert_equal
from proboscis.asserts import assert_false
from proboscis.asserts import assert_raises
from proboscis.asserts import assert_true
from proboscis import after_class
from proboscis import before_class
from proboscis import SkipTest
from proboscis import test
db_config = {
"url": "test.db.mycompany.com",
"user": "service_admin",
"password": "pass"
}
@test
def initialize_database():
"""Creates a local database."""
mymodule.create_database()
assert_true(mymodule.tables_exist())
@test(depends_on=[initialize_database])
def initialize_web_server():
"""Starts up the web service."""
mymodule.start_web_server()
admin = mymodule.get_admin_client()
assert_true(admin.service_is_up)
class UserTests(object):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"normal"
}
return new_user_config
@before_class
def create_user(self):
"""Create a user."""
random.seed()
global test_user
test_user = None
new_user_config = self.generate_new_user_config()
admin = mymodule.get_admin_client()
self.test_user = admin.create_user(new_user_config)
assert_equal(self.test_user.username, new_user_config["username"])
assert_true(self.test_user.id is not None)
assert_true(isinstance(self.test_user.id, int))
@after_class(always_run=True)
def delete_user(self):
if self.test_user is None:
raise SkipTest("User tests were never run.")
admin = mymodule.get_admin_client()
admin.delete_user(self.test_user.id)
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'password'})
def cant_login_with_wrong_password(self):
assert_raises(mymodule.UserNotFoundException, mymodule.login,
{'username':self.test_user.username, 'password':'blah'})
@test
def successful_login(self):
self.client = mymodule.login({
'username':self.test_user.username, 'password':'password'})
@test(depends_on=[successful_login])
def change_profile_image(self):
"""Test changing a client's profile image."""
assert_equal("default.jpg", self.client.get_profile_image())
self.client.set_profile_image("spam.jpg")
assert_equal("spam.jpg", self.client.get_profile_image())
@test(groups=["user", "service.tests"])
class NormalUserTests(UserTests):
@test(depends_on=[UserTests.successful_login])
def a_normal_user_cant_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.create_user,
self.generate_new_user_config())
@test(depends_on=[UserTests.successful_login])
def a_normal_user_cant_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
assert_raises(mymodule.AuthException, self.client.delete_user,
self.test_user.id)
@test(groups=["user", "service.tests"])
class AdminUserTests(UserTests):
@staticmethod
def generate_new_user_config():
"""Constructs the dictionary needed to make a new user."""
new_user_config = {
"username": "TEST_%s_%s" % (datetime.now(), random.randint(0, 256)),
"password": "password",
"type":"admin"
}
return new_user_config
@test(depends_on=[UserTests.successful_login])
def an_admin_user_can_create_users(self):
"""Make sure the given client cannot perform admin actions.."""
self.new_user = self.client.create_user(self.generate_new_user_config())
# Make sure it actually logs in.
self.new_user_client = mymodule.login({
'username':self.new_user.username, 'password':'password'})
@test(depends_on=[an_admin_user_can_create_users])
def an_admin_user_can_delete_users(self):
"""Make sure the given client cannot perform admin actions.."""
self.client.delete_user(self.new_user.id)
# Add more tests in the service.tests group here, or in any other file.
# Then when we're finished...
@test(groups=["service.shutdown"], depends_on_groups=["service.tests"],
always_run=True)
def shut_down():
"""Shut down the web service and destroys the database."""
admin = mymodule.get_admin_client()
if admin.service_is_up:
mymodule.stop_web_server()
assert_false(admin.service_is_up())
mymodule.destroy_database()
Its possible to create empty test entries that link groups together using the proboscis.register function without a class or function. A good place to do (as well as store other bits of configuration) is in the start up script you write for Proboscis. Here’s an example:
from proboscis import register
from proboscis import TestProgram
def run_tests():
from tests import service_tests
# Now create some groups of groups.
register(groups=["fast"], depends_on_groups=["unit"])
register(groups=["integration"],
depends_on_groups=["service.initialize",
"service.tests",
"service.shutdown"])
register(groups=["slow"],
depends_on_groups=["fast", "integration"])
# Run Proboscis and exit.
TestProgram().run_and_exit()
if __name__ == '__main__':
run_tests()
Here the groups “fast”, “integration”, and “slow” are created as simple dependencies on other groups. This makes it possible to, for example, run all “slow” tests with the following command:
python runtests.py --group=slow