Testing statecharts with statecharts¶

Like software, statecharts can be tested too.

It is always possible to test the execution of a statechart by hand. The simulator stores and returns several values that can be inspected during the execution, including the active configuration, the list of entered or exited states, etc. The functional tests in tests/test_simulator.py on the GitHub repository are several examples of this kind of tests.

However, this approach is not really pleasant to test statecharts, and even less when it comes to specify invariants or behavioral conditions.

Thanks to PySS, module pyss.testing makes it easy to test statecharts. In particular, this module brings a way to test statecharts using... statecharts!

How to write tests?¶

Remark: in the following, the term tested statechart refers to the statechart that will be tested, while the term statechart testers (or simply testers) refers to the ones that express conditions or invariants that should be satisfied by the tested statechart.

Statechart testers are classical statechart, in the sense that their syntax nor their semantic differ from tested statecharts. The main difference comes from the events they receive, and from the variables and functions exposed in their execution context (see Built-in Python code evaluator).

Specific received events¶

In addition to (optional) internal events, a statechart tester is expected to automatically receive a deterministic sequence of the three following events:

start – this event is sent when a stable initial state is reached by the tested statechart.

stop – this event is sent when the execution of the tested statechart ends (either because it reaches a final configuration, or no more transition can be processed, or because its execution was interrupted by stop(), see below).

step – this event is sent after the computation and the execution of a MacroStep in the tested statechart.

Specific contextual data¶

Each tester is executed using by default a PythonEvaluator and as such, contextual data (the context) are available during execution. In the case of a tester, this context is always populated and updated with the following items:

Moreover, the context of a tester also expose the context of the tested statechart through the key context. This way, you can define guards (or actions) that rely on the data available in the tested statechart.

For example, the guard in the following transition (in a statechart tester) accesses the destination value of its tested statechart context.

statemachine:
    # ...
    transitions:
     - guard: context['destination'] > 0
       # ...

Specific expected behavior¶

It is expected that your testers ends in a final configuration (ie. all the leaves of the active configuration are final states) when a valid execution of a tested statechart ends.

This is very important, and this is why its deserves a dedicated subsection.

At the end of the execution of a test, an assertion will be raised if there is a tester which is not in a final configuration.

Examples¶

The following examples are relative to this statechart.

Destination is finally reached¶

This tester is an example of a test that needs to end in a final configuration. It ensures that a destination is always reached before the end of the execution of the elevator statechart.

The state waiting awaits that a floorSelected event is processed. When the floor is selected, it waits until current == destination to go in destinationReached state. If the execution ends (stop event) before the destination is reached (ie. in another state than destinationReached), the tester execution does not end in a final state, meaning that the test fails.

statechart:
  name: Destination is reached
  initial: waiting
  states:
    - name: waiting
      transitions:
        - target: floorSelected
          event: step
          guard: processed('floorSelected')  # Avoid consumed('floorSelected') here, as it does not imply its processing
    - name: floorSelected
      transitions:
        - target: destinationReached
          event: step
          guard: context['current'] == context['destination']
    - name: destinationReached
      transitions:
        - target: final
          event: stop
        - target: floorSelected
          event: step
          guard: processed('floorSelected')
    - name: final
      type: final

Doors are closed while moving¶

This tester is an example of a test that raises an AssertionError. It checks that the elevator can not move (ie. be in moving state) while the doors are opened. If this happens, a transition to error occurs. The on entry of error then raises an AssertionError.

statechart:
  name: Doors are closed while moving
  initial: start
  states:
    - name: start
      initial: check_doors
      transitions:
        - target: ok  # If nothing bad happens, then we should go in a final state
          event: stop
      states:
        - name: check_doors  # Check if the doors are initially opened or not
          transitions:
            - target: opened
              guard: not context['doors'].opened
            - target: closed
              guard: context['doors'].opened
        - name: opened
          transitions:
            - target: closed
              event: step
              guard: not context['doors'].opened
            - target: error  # If we are moving and the doors are opened, error!
              event: step
              guard: active('moving')
        - name: closed
          transitions:
            - target: opened
              event: step
              guard: context['doors'].opened
    - name: error
      on entry: assert False, 'doors not closed and elevator is moving'
    - name: ok
      type: final

7th floor is never reached¶

This example shows that assertion can be made on transition action too. This dummy example could fail if the current floor is 7. You could use it to test what happens when a test fails.

statechart:
  name: Never go to 7th floor
  initial: waiting
  states:
    - name: waiting
      transitions:
        - event: step
          guard: context['current'] == 7
          action: |
            assert False, 'Should not go on 7th!'  # We can make assertion in action too!
        - target: ok
          event: stop
    - name: ok
      type: final

The testing module¶

The pyss.testing module essentially defines the following classes:

TesterConfiguration defines the configuration of a test

StateChartTester initializes a test.

class pyss.testing.TesterConfiguration(statechart: pyss.model.StateChart, evaluator_klass=None, simulator_klass=None)¶

A tester configuration mainly serves as a data class to prepare tests. Such a configuration remembers which is the statechart to test, using which evaluator and which simulator.

Parameters:

statechart – A model.StateChart instance
evaluator_klass – An optional callable (eg. a class) that takes no input and return a evaluator.Evaluator instance that will be used to initialize the simulator.
simulator_klass – An optional callable (eg. a class) that takes as input a model.StateChart instance and an optional evaluator.Evaluator instance, and return an instance of simulator.Simulator (or anything that acts as such).

add_test(statechart: pyss.model.StateChart, evaluator_klass=None, simulator_klass=None)¶

Add the given statechart as a test.

Parameters:

statechart – A model.StateChart instance.
evaluator_klass – An optional callable (eg. a class) that takes no input and return a evaluator.Evaluator instance that will be used to initialize the simulator.
simulator_klass – An optional callable (eg. a class) that takes as input a model.StateChart instance and an optional evaluator.Evaluator instance, and return an instance of simulator.Simulator (or anything that acts as such).

build_tester(events: list) → pyss.testing.StateChartTester¶

Build a StateChartTester from current configuration.

Parameters:	events – A list of `model.Events` instances that serves as a scenario
Returns:	A `StateChartTester` instance.

class pyss.testing.StateChartTester(simulator: pyss.simulator.Simulator, testers: list, events: list)¶

A tester with given simulator (that simulates the statechart to test), a list of testers (simulators that runs tests) and a list of events (scenario).

Parameters:	simulator – The simulator containing the statechart to test testers – A list of `simulator.Simulator` containing the tests events – A list of `model.Event` that represents a scenario

execute(max_steps=-1) → list¶

Repeatedly calls self.execute_once() and return a list containing the returned values of self.execute_once().

Raises AssertionError:
Parameters:	max_steps – An upper bound on the number steps that are computed and returned. Default is -1, no limit. Set to a positive integer to avoid infinite loops in the statechart simulation.
Returns:	A list of `MacroStep` instances produced by the underlying simulator.
	if simulation terminates and a tester is not in a final configuration

execute_once() → pyss.simulator.MacroStep¶

Call execute_once() on the underlying simulator, and consequently call execute() on the testers.

Raises AssertionError:
Returns:	The `simulator.MacroStep` instance returned by the underlying call.
	if simulation terminates and a tester is not in a final configuration

stop()¶

Stop the execution and raise an AssertionError if at least one tester is not in a final configuration.

Raises AssertionError:
	if a tester is not in a final configuration

Executing tests¶

In order to test a statechart, you need to get:

A statechart you want to test.

At least one tester, ie. a statechart that checks some invariant or condition.

A test scenario, which is in fact a list of event instances.

Assume we previously defined and imported a tested statechart tested_sc and a tester tester, two instances of StateChart.

We first define a test configuration.

from pyss.testing import TesterConfiguration, StateChartTester

config = TesterConfiguration(tested_sc)

This configuration is mainly used to set a test environment (the setUp() of a unit test). We can specify which code evaluator will be used, by specifying a callable that return an Evaluator instance. Remember that in Python, a class is a callable, so it is perfectly legit to write this:

from pyss.evaluator import DummyEvaluator

config = TesterConfiguration(tested_sc, evaluator_klass=DummyEvaluator)

It is also possible to specify a different semantic for the execution of the tested statechart (see Implementing other semantics). This can be done using the simulator_klass parameter.

config = TesterConfiguration(tested_sc, simulator_klass=MyOtherSimulatorClass)

This way, you can for example test that your initial statechart is invariant under several distinct simulator. It is now time to specify which are the testers we want to use. Not surprisingly, add_test() method does the job. This method takes a StateChart instance that is a statechart tester.

config.add_test(tester_sc)

You could add as many testers as you want. It is also possible to provide specific code evaluator and specific simulator for each tester, as it was the case for the tested statechart. The syntax is the same:

config.add_test(other_tester_sc, evaluator_klass=DummyEvaluator, simulator_klass=MyOtherSimulatorClass)

Our test configuration is now ready, and we can go one step further. We now need to create a test. This can be done using build_tester(). This method takes a list of Event which will be sent to the tested statechart. This list can be viewed as a scenario for the test. The method returns an instance of StateChartTester.

events = [Event('event1'), Event('event2'), Event('event3')]
test = config.build_tester(events)

Using a test, you can execute the test by calling execute_once() or execute() (which repeatedly calls the first one). A test mainly executes the tested statechart, step by step, and sends specific events to the tester. The tester are then executed (using the execute() method of the simulator).

A test is considered as a success if the call to execute() ends, and no AssertionError was raised.

Depending on the underlying simulator (but this at least concerns the default one!), the execution of a statechart can be infinite. As for simulator’s execute(), you can specify a max_steps parameter to limit the number of steps that are executed.

test.execute(max_steps=10)

At the end of the execution, you must call the stop() method. This method sends a stop event to the statechart testers, and checks whether they are all in a final configuration.

As a shortcut, StateChartTester exposes a context manager that does the job for you. This context manager can be used as follows:

with config.build_tester(events) as test:
    test.execute()

A test fails when one of the following occurs:

An AssertionError is raised by one of the statechart testers.

There is at least one tester that is not in a final configuration when stop() is called (or when the context manager is exited).

Integrating with unittest¶

It is very easy to use the testing module with Python unittest.

Consider the source of tests/test_testing.py:

import unittest
from pyss import io
from pyss.model import Event
from pyss.testing import TesterConfiguration


class ElevatorTests(unittest.TestCase):
    def setUp(self):
        self.sc = io.import_from_yaml(open('examples/concrete/elevator.yaml'))
        self.tests = {
            'destination_reached': io.import_from_yaml(open('examples/tester/elevator/destination_reached.yaml')),
            'closed_doors_while_moving': io.import_from_yaml(open('examples/tester/elevator/closed_doors_while_moving.yaml')),
            'never_go_7th_floor': io.import_from_yaml(open('examples/tester/elevator/never_go_7th_floor.yaml')),
        }
        self.scenarios = {
            '4th floor': [Event('floorSelected', data={'floor': 4})],
            '7th floor': [Event('floorSelected', data={'floor': 7})],
        }
        self.tester = TesterConfiguration(self.sc)

    def test_destination_reached(self):
        self.tester.add_test(self.tests['destination_reached'])
        with self.tester.build_tester(self.scenarios['4th floor']) as tester:
            tester.execute()

    def test_doors_closed_while_moving(self):
        self.tester.add_test(self.tests['closed_doors_while_moving'])
        with self.tester.build_tester(self.scenarios['4th floor']) as tester:
            tester.execute()

    def test_never_go_underground(self):
        self.tester.add_test(self.tests['never_go_7th_floor'])

        with self.assertRaises(AssertionError):
            with self.tester.build_tester(self.scenarios['7th floor']) as tester:
                tester.execute()