Examples¶
Examples showing how to use this package.
Getting Started¶
The simplest way to test your API is just to ask swaggerconformance
to validate it as follows (assuming your swagger spec is hosted at
http://example.com/api/schema.json
- a local file path could be used
instead):
from swaggerconformance import api_conformance_test
api_conformance_test('http://example.com/api/schema.json')
This will simply make requests against your API using data generated to match your API schema, and ensure that the response codes and data returned also match your schema.
Handling Failures¶
If this testing fails, you’ll be shown an example input to an API operation which resulted in a response that didn’t match your schema. Something like:
Falsifying example: single_operation_test(
client=SwaggerClient(schema_path='http://example.com/api/schema.json'),
operation=OperationTemplate(
method='get',
path='/example/{exint}',
params={'exint': ParameterTemplate(name='exint', type='integer', required=True)}),
params={'exint': -1})
...
<Some stack trace here>
...
AssertionError: Response code 500 not in {200, 404}
This shows that when testing the GET
operation on the endpoint
/example/{exint}
, using the value -1
for exint
resulted in a
500
response code from the server, which is not one of the
documented response codes (and any 5XX
response code is clearly an
error!). From this we can tell that this server isn’t handling negative
numbers for this parameter properly and needs to be made more robust.
As an aside, one great feature of hypothesis is once it finds a failing
example, it will simplify it down as far as it can. So here it might
have first found that -2147483648
failed, but instead of just
reporting that and let you figure out if that number is special, it
tries to find the simplest failing input value, e.g. here reducing down
to simply -1
.
Targeted and Stateful Testing¶
The basic testing above just throws data at your API, but it’s useful to be able to target specific endpoints, or build sequenced tests where the output of one operation is used as the input to another.
Example¶
Here’s a small example of a test that creates a resource with a PUT
containing some generated data, and then verifies that a GET
returns
the exact same data.
You can run this test yourself by starting the datastore_api.py
server and running the ex_targeted_test.py
script in the examples
directory.
A walkthrough of the code there follows.
Walkthrough¶
The first part of the setup involves creating a client that will make requests against the API, and creating templates for all operations and parameters the API exposes. The client can be given a URL or local file path to read the schema from.
import hypothesis
import swaggerconformance
# Create the client to access the API, and templates for the API operations.
schema_url = 'http://127.0.0.1:5000/api/schema'
client = swaggerconformance.client.SwaggerClient(schema_url)
api_template = swaggerconformance.apitemplates.APITemplate(client)
The next step pulls out the operations that will be tested, and creates
hypothesis
strategies for generating inputs to them. Here specific
operations are accessed, but it’s possible to just iterate through all
operations in api_template.template_operations()
. The
value_factory
generates strategies for individual data types being
used - we’ll just use the defaults, but the next section covers adding
your own extra data types.
# Get references to the operations we'll use for testing, and strategies for
# generating inputs to those operations.
value_factory = swaggerconformance.valuetemplates.ValueFactory()
put_operation = api_template.endpoints["/apps/{appid}"]["put"]
put_strategy = put_operation.hypothesize_parameters(value_factory)
get_operation = api_template.endpoints["/apps/{appid}"]["get"]
get_strategy = get_operation.hypothesize_parameters(value_factory)
That’s all the setup done - now to write the hypothesis
test. The
hypothesis
docs
go through details of doing this and the available test
settings.
In short though, you write a function which validates the property of
your API you want to verify, and through a decorator hypothesis
runs
this function multiple times. Each attempt uses different parameter
values chosen to test all corners of the allowed values - the more
attempts you allow hypothesis
, the more thorough it can be and the
greater the chance it has to flush out bugs.
# Hypothesis will generate values fitting the strategies that define the
# parameters to the chosen operations. The decorated function is then called
# the chosen number of times with the generated parameters provided as the
# final arguments to the function.
@hypothesis.settings(max_examples=200)
@hypothesis.given(put_strategy, get_strategy)
def single_operation_test(client, put_operation, get_operation,
put_params, get_params):
"""PUT a new app with some data, then GET the data again and verify it
matches what was just sent in."""
# Use the client to make a request with the generated parameters, and
# assert this returns a valid response.
response = client.request(put_operation, put_params)
assert response.status in put_operation.response_codes, \
"{} not in {}".format(response.status, put_operation.response_codes)
# The parameters are just dictionaries with string name keys and correct
# format values.
# We generated two independent sets of parameters, so replace the id in the
# second set with the id from the first so they reference the same object.
get_params["appid"] = put_params["appid"]
response = client.request(get_operation, get_params)
# `response.data` contains the JSON body of the response converted to
# Python objects.
# `payload` is always the name of body parameters on requests.
out_data = response.data.data
in_data = put_params["payload"]["data"]
assert out_data == in_data, "{!r} != {!r}".format(out_data, in_data)
# Finally, remember to call the test function to run the tests, and that
# hypothesis provides the generated parameter arguments.
single_operation_test(client, put_operation, get_operation)
Custom Data Types¶
swaggerconformance
supports generating values for all the standard
datatypes from the OpenAPI
schema.
However you might have defined your own more specific ones and want to
generate instances of them to get more thorough testing of valid inputs
to your API.
Example¶
For example, suppose you have an API operation that takes a colour as a parameter, so you’ve defined a new datatype for your API as:
Common Name | type |
format |
Comments |
---|---|---|---|
colour | string |
hexcolour |
# and six hex chars, e.g. #3FE7D9 |
By default, clearly swaggerconformance
won’t know what this
format
is, and will just generate string
type data as input for
parameters of this format. So for most requests, your API will just
rejecting them with some 4XX
response because the parameter isn’t of
the correct format. You might prefer that valid hex colours were being
generated to test a greater number of successful API requests. New
type
and format
support can be added into swaggerconformance
fairly easily.
The first step here is to create a ValueTemplate
which can build a
hypothesis strategy to generate these hexcolour
values:
import string
import hypothesis.strategies as hy_st
from swaggerconformance import valuetemplates
class HexColourTemplate(valuetemplates.ValueTemplate):
"""Template for a hex colour value."""
def __init__(self, enum=None):
# Various parameters like length don't make sense for this field, but
# an enumeration of valid values may still be provided.
super().__init__()
self._enum = enum
def hypothesize(self):
# If there's a specific set of allowed values, the strategy should
# return one of those.
if self._enum is not None:
return hy_st.sampled_from(self._enum)
# We'll generate values as strings, but we could generate integers and
# convert those to hex instead.
strategy = hy_st.text(alphabet=set(string.hexdigits),
min_size=6,
max_size=6)
# Don't forget to add the leading `#`.
strategy = strategy.map(lambda x: "#" + x)
return strategy
New ValueTemplate
s just needs to provide a hypothesize(self)
method which returns a hypothesis
strategy - __init__
is
optional if no parameters are needed.
You can see the hypothesis docs for more information on building up strategies. One written, you can test values your template will produce - e.g.:
>>> template = HexColourTemplate()
>>> strategy = template.hypothesize()
>>> strategy.example()
'#CafB0b'
Now that the template for values of this type is defined, we just need
to create an updated factory that will generate them. To do this,
inherit from the built in one, and add logic into an overridden
create_value
method:
import swaggerconformance
class HexColourValueFactory(swaggerconformance.valuetemplates.ValueFactory):
def create_value(self, swagger_definition):
"""Handle `hexcolour` string format, otherwise defer to parent class.
:type swagger_definition:
swaggerconformance.apitemplates.SwaggerParameter
"""
if (swagger_definition.type == 'string' and
swagger_definition.format == 'hexcolour'):
return HexColourTemplate(swagger_definition.enum)
else:
return super().create_value(swagger_definition)
Now whenever creating strategies for generating parameters for
operations, use the new factory. Then anytime string
, hexcolour
is the datatype of a parameter, the new template will be used to
generate a strategy for it. So in the example code in the previous
section, the change would just be:
schema_url = 'http://example.com/api/schema.json'
client = swaggerconformance.client.SwaggerClient(schema_url)
api_template = swaggerconformance.apitemplates.APITemplate(client)
value_factory = HexColourValueFactory() # Use enhanced factory for values.
put_operation = api_template.endpoints["/apps/{appid}"]["put"]
put_strategy = put_operation.hypothesize_parameters(value_factory)
...
There’s no reason that the factory couldn’t check other properties of
the parameter to decide what type of values to generate, e.g. the
name
- but remember that likely clients won’t have whatever special
logic you add here so it may be better to assign a special data type
instead.
If needed, you can replace any of the built in value templates - e.g. to restrict the alphabet of all generated basic strings - but again remember that making the generation more restrictive might mean bugs are missed if clients don’t have the same restrictions.