Tutorial

This chapter covers the whole Tutorial about radish and it’s features.

Feature files

All tests are written in so-called feature files. Feature files are plain text files ending with .feature. A feature file can contain only one BDD Feature written in a natural language format called Gherkin. However, radish is able to run one or more feature files. The feature files can be passed to radish as arguments:

radish features/
radish features/SumNumbers.feature features/DivideNumbers.feature
radish features/unit features/functional

Feature

A Feature is the main part of a feature file. Each feature file must contain exactly one Feature. This Feature should represent a test for a single feature in your software similar to a test class in your unit code tests. The Feature is composed of a Feature sentence and a Feature description. The feature sentence is a short precise explanation of the feature which is tested with this Feature. The feature description as a more verbose explanation of the feature which is tested. There you can answer the Why and What questions. A Feature has the following syntax:

Feature: <Feature sentence>
    ... Feature description
    on multiple lines ...

A Feature must contain one or more Scenarios which are run when this feature file is executed.

Feature: <Feature sentence>
    ... Feature description
    on multiple lines ...

    Scenario: <Scenario 1 sentence>
        ... Steps ...

    Scenario: <Scenario 2 sentence>
        ... Steps ...

Scenario

A Scenario is located inside a Feature. You can think of a Scenario as of a standalone test case for the feature you want to test. A Scenario contains one or more Steps. Each Scenario must have a unique sentence inside a Feature.

Feature: My Awesome Feature
    In order to document
    radish I write this feature.

    Scenario: Test feature
        ... Some Steps ...

    Scenario: Test feature with a bad case test
        ... Some Steps ...

Scenario Outline

A Scenario Outline is a more advanced version of a standard Scenario. It allows you to run a Scenario multiple times with different input values. A Scenario Outline is defined with Examples. The Scenario is run with the input data from each Example. The data from the Example can be accessed in a Scenario with the name of the data inside < and >. For example see the following Scenario Outline which divides multiple numbers from the Examples:

Feature: Test dividing numbers
    In order to test the
    Scenario Outline features of
    radish I test dividing numbers.

    Scenario Outline: Divide Numbers
        Given I have the number <number1>
        And I have the number <number2>
        When I divide them
        Then I expect the result to be <result>

    Examples:
        | number1 | number2 | result |
        | 10      | 2       | 5      |
        | 6       | 3       | 2      |
        | 24      | 8       | 3      |

Scenario Loop

A Scenario Loop is a standard Scenario which is repeated for a given amount of iterations. Scenario Loops can often be useful when stabilization tests are performed in a CI environment. Scenario Loops have the following syntax:

Feature: My Awesome Feature
    In order to document
    radish I write this feature.

    Scenario Loop 10: Some stabilization test
        ... Some Steps ...

Note: Scenario Loops are not standard gherkin

Scenario Precondition

Sometimes it can be very useful to reuse specific Scenarios. That’s why we’ve decided to implement Scenario Preconditions in radish even though it’s not common for a BDD tool. Before you start using Scenario Preconditions you should really think about the reason why you are using it. Behavior Driven Development Scenarios should be as short and concise as possible without a long list of dependencies. But there will always be these edge cases where it really makes sense to have a precondition for your Scenario. Every Scenario can be used as a Precondition Scenario. Scenario Preconditions are implemented as special tags:

Feature: My Awesome Feature
    In order to document
    radish I write this feature.

    @precondition(SomeFeature.feature: An awesome Scenario)
    Scenario: Do some crazy stuff
        When I add the following users to the database
            | Sheldon | Cooper |
        Then I expect to have 1 user in the database

radish will import the Scenario with the sentence An awesome Scenario from the feature file SomeFeature.feature and run it before the Do some crazy stuff Scenario. The following lines will be written:

Feature: My Awesome Feature
    In order to document
    radish I write this feature.

    @precondition(SomeFeature.feature: An awesome Scenario)
    Scenario: Do some crazy stuff
      As precondition from SomeFeature.feature: An awesome Scenario
        Given I setup the database
      From scenario
        When I add the following users to the database
            | Sheldon | Cooper |
        Then I expect to have 1 user in the database

As you can see radish will print some information about the Scenario where the Steps came from. radish supports multiple and nested Scenario Preconditions, too. Recursions are detected and radish will print an appropriate error message.

If you have preconditions in a Scenario it’s inconvenient to send it to your colleague or post it somewhere because you have multiple files. radish is able to resolve all preconditions and expand them to a single file. Use the radish show --expand command to do so:

$ radish show --expand MyFeature.feature
Feature: My Awesome Feature
    In order to document
    radish I write this feature.

    #@precondition(SomeFeature.feature: An awesome Scenario)
    Scenario: Do some crazy stuff
        Given I setup the database
        When I add the following users to the database
            | Sheldon | Cooper |
        Then I expect to have 1 user in the database

The information about the precondition is commented out.

Note: Scenario Preconditions are not standard gherkin

Background

A Background is a special case of the Scenario. It’s used to add some context to each Scenario of the same Feature. You can think of it as a setup Scenario for the other Scenarios. It consists of one or more Steps in exactly the same way as regular Scenarios. The Background is run after the before hooks of each Scenario but before the Steps of this Scenario.

To following example illustrates a possible use for a Background:

Feature: Restricted site support
    As a user of AwesomeSite
    I want to restrict my personal sites
    to specific users.

    Background: Have a multi user setup
        Given a user named Bruce
        And a user named Peter
        And a user named Tony
        And a personal site owned by Bruce

    Scenario: Grant access to personal site
        Given Bruce grants access to Tony
        When I'm logged in as Tony
        Then I can access Bruce personal site

    Scenario: Deny access to personal site
        Given Bruce grants access to Tony
        When I'm logged in as Peter
        Then I cannot access Bruce personal site

Note: the entire example can be found here.

Cucumber defined some useful good practices for using backgrounds. It’s worth to read them carefully.

Steps

The steps are the heart piece of every Feature file. A line in a Scenario is called Step. The steps are the only thing which are really executed in a test. A Step is written in a human readable language. Each step is parsed by radish and matched with a step implementation written in python. If a Step does not match any step implementation radish will raise an exception and abort the run.

All steps are implemented in python files located inside the radish basedirs. Per default this base directory points to $PWD/radish. However, the base directory location can be changed by specifying the -b option when triggering radish. You can also specify -b multiple times to load from multiple locations. There are several ways how to implement steps. The most common way is by decorating your step implementation functions with one of the following decorators:

  • @step(pattern)
  • @given(pattern)
  • @when(pattern)
  • @then(pattern)

The difference between those four decorators is that for the given, when and then decorator the corresponding keyword is prefixed. For example @given("I have the number") becomes the pattern Given I have the number.

A basic steps.py file with some step implementations could look like the following:

# -*- coding: utf-8 -*-

from radish import given, when, then

@given("I have the number {number:g}")
def have_number(step, number):
    step.context.numbers.append(number)


@when("I sum them")
def sum_numbres(step):
    step.context.result = sum(step.context.numbers)


@then("I expect the result to be {result:g}")
def expect_result(step, result):
    assert step.context.result == result

The first example of a step implementation function is always an object of type Step.

Another way to implement step functions is using an enitre class:

# -*- coding: utf-8 -*-

from radish import steps

@steps
class Calculator(object):
    def have_number(self, step, number):
        """I have the number {number:g}"""
        step.context.numbers.append(number)

    def sum_numbres(self, step):
        """I sum them"""
        step.context.result = sum(step.context.numbers)

    def expect_result(self, step, result):
        """I expect the result to be {result:g}"""
        assert step.context.result == result

With the @steps decorator all methods of the given class are registered as steps. The step pattern is always the first line of the docstring of each method. If a method inside the call is not a step implementation you can add the method name to the ignore attribute of this class:

# -*- coding: utf-8 -*-

from radish import steps

@steps
class Calculator(object):

    ignore = ["validate_number"]

    def validate_number(self, number):
        """Validate the given number"""
        ...

    def have_number(self, step, number):
        """I have the number {number:g}"""
        self.validate_number(number)
        step.context.numbers.append(number)

Step Pattern

The pattern for each Step can be defined in two ways. The default ways is to specify the Step pattern in a format similar to the one used by python’s str.format() method. The pattern can be a simple string:

@given("I sum all my numbers")
...

This Step pattern doesn’t have any arguments. To specify arguments use the {ARGUMENT_NAME:ARGUMENT_TYPE} format:

@given("I have the number {number:g}")
def have_number(step, number):
    ...

The argument will be passed as keyword argument to the step implementation function with the specified name. If no name is specified the arguments are positional:

@given("I have the numbers {:g} and {:g}")
def have_numbers(step, number1, number2):
    ...

Per default the following argument types are supported:

Type Characters matched Output type
w Letters and underscore str
W Non-letter and underscore str
s Whitespace str
S Non-whitespace str
d Digits (effectively integer numbers) int
D Non-digit str
n Numbers with thousands separators (, or .) int
% Percentage (converted to value/100.0) float
f Fixed-point numbers float
e Floating-point numbers with exponent e.g. 1.1e-10, NAN (all case insensitive) float
g General number format (either d, f or e) float
b Binary numbers int
o Octal numbers int
x Hexadecimal numbers (lower and upper case) int
ti ISO 8601 format date/time e.g. 1972-01-20T10:21:36Z (“T” and “Z” optional) datetime
te RFC2822 e-mail format date/time e.g. Mon, 20 Jan 1972 10:21:36 1000 datetime
tg Global (day/month) format date/time e.g. 20/1/1972 10:21:36 AM 1:00 datetime
ta US (month/day) format date/time e.g. 1/20/1972 10:21:36 PM 10:30 datetime
tc ctime() format date/time e.g. Sun Sep 16 01:03:52 1973 datetime
th HTTP log format date/time e.g. 21/Nov/2011:00:07:11 +0000 datetime
ts Linux system log format date/time e.g. Nov 9 03:37:44 datetime
tt Time e.g. 10:21:36 PM -5:30 time
MathExpression Mathematic expression containing: [0-9 +-*/%.e]+ float

Radish provides a way to extend these types. This could be useful to directly inject more advanced objects to the step implementations:

from radish import arg_expr

@arg_expr("User", r"[A-Z][a-z]+ [A-Z][a-z]+")
def user_argument_expression(text):
    """
        Return a user object by the given name
    """
    if text not in world.database.users:  # no user found
        return None

    return world.database.users[text]

This argument type can be used like this in the Step pattern:

from radish import then

@then("I expect the user {user:User} has the email {}")
def expect_user_has_email(step, user, expected_email):
    assert user.email == expected_email, "User has email '{0}'.
       Expected was email '{1}'".format(user.email, expected_email)

If these Step patterns do not fit all your use cases you could use your own Regular Expression to match a Step sentence:

from radish import then

@then(re.compile(r"I expect the user ([A-Z][a-z]+ [A-Z][a-z]+|PENNY&LEONARD)+"))
def complex_stuff(step, user):
    ...

The groups matched by the Regular Expression are passed to the step implementation function.

Step Behave like

Sometimes it could be useful to call another step within a step. For example it could be useful if you want to change the interface but still support the old steps or if you want to combine multiple steps in one step. This feature is called behave like and you can use it as the following:

@step("I want to setup the database")
def setup_database(step):
    step.behave_like("I start the database server")
    step.behave_like("I add the system users to the database")
    step.behave_like("I add all roles to the database")

Step Tables

Step Tables are used to provide table-like data to a Step. The Step Table syntax looks similar to the Scenario Outline Examples:

...
Scenario: Check database
    Given I have the following users
       | Peter | Parker | Spiderman |
       | Bruce | Wayne  | Batman    |
    When I add them to the database
    Then I expect 2 users in the database

The Step Table can be accessed in the Step Implementation function through the step.table attribute which is a list of lists:

# -*- coding: utf-8 -*-

from radish import given, when, then

@given("I have the following users")
def have_number(step):
  step.context.users = step.table


@when("I add them to the database")
def sum_numbres(step):
  for user in step.context.users:
      step.context.database.users.add(forename=user[0], \
          lastname=user[1], nickname=user[2])


@then("I expect {number:g} users in the database")
def expect_result(step, number):
  assert len(step.context.database.users) == number

Step Text data

Like the Step Tables a Step can also get an arbitrary text block as input. The syntax to pass text data to a Step looks like this:

...
Scenario: Test quote system
   Given I have the following quote
       """
       To be or not to be
       """
   When I add it to the database
   Then I expect 1 quotes in the database

To access this text data you can use the text attribute on the step object:

# -*- coding: utf-8 -*-

from radish import given, when, then

@given("I have the following quote")
def have_quote(step):
    step.context.quote = step.text


@when("I add it to the database")
def add_quote_to_db(step):
        step.context.database.quotes.append(step.context.quote)


@then("I expect {number:g} quote in the database")
def expect_amount_of_quotes(step, number):
    assert len(step.context.database.quotes) == number

Tags

Tags are a way to group or classify Features and Scenarios. Radish is able to only run Features or Scenarios with specific Tags. Tags are declared with a similar syntax as decorators in python:

@regression
Feature: Some important feature
   In order to demonstrate
   the Tag feature in radish
   I write this feature.

   @good_case
   Scenario: Some good case test
       ...

   @bad_case
   Scenario: Some bad case test
       ...

Note: a Scenario inherits all tags of the Feature it is defined in!

When triggering radish you can pass the --tags command line option followed by a tag expression. Tag expressions are parsed with tag-expressions. Only these Features/Scenarios are ran.

Run all regression tests:

radish features/ --tags regression

Run all good case or bad case tests:

radish features/ --tags 'good_case or bad_case'

Constants

Constants are specific Tags which define a constant which can be used in the Steps. This could be useful when you have values which are used in several points in a Feature and which should be named instead of appear as magic numbers. A sample use-case I’ve seen is specifying a base temperature:

@constant(base_temperature: 70)
Feature: Test heater
   In order to test my
   heater system I write
   the following scenarios.

   Scenario: Test increasing the temperature
       Given I have the room temperature ${base_temperature}
       When I increase the temperature about 5 degrees
       Then I expect the temperature to be ${base_temperature} + 5

Note: Constants are not standard gherkin

Terrain and Hooks

In addition to the Step implementation radish provides the possibility to implement hooks. These hooks are usually placed in a file called terrain.py inside the base directory. Hooks can be used to setup and tear down the Features or Scenarios. There are two different hook objects:

  • before
  • after

These can be combined with the following hook subjects:

  • all
  • each_feature
  • each_scenario
  • each_step

Hooks are simply registered by adding these hook objects and subjects as decorators to python functions:

# -*- coding: utf-8 -*-

from radish import before

from database import Database

@before.each_scenario
def connect_database(scenario):
   scenario.context.database = Database(name="foobar")
   scenario.context.database.connect()

The python functions must accept the respective object and in the case of all a second argument which is the radish run marker (a unique run id):

# -*- coding: utf-8 -*-

from radish import after

@after.all
def connect_database(features, marker):
   scenario.context.database.disconnect()

The hooks are called in the order of registration.

If you are using Tags you can specify that a certain hook is only called for Features, Scenarios or Steps with the according tags.

from radish import after

@after.each_scenario(on_tags='bad_case or crash')
def cleanup(scenario):
    # do some heavy cleanup!
    pass

Contexts

As you may have noticed: each Feature and Scenario has it’s own context. You can dynamically add attributes to this context. All Steps in a Scenario have the same context. This is the preferred way to share data between steps over the world object.

from radish import before, given

@given("I have the number {number:g}")
def have_number(step, number):
   # accessing Scenario specific context
   step.context.number = number


@before.each_feature
def setup(feature):
    # accessing Feature specific context
    feature.context.setup = True

World

The world is a “global” radish context. It is used by radish to store the configuration and other utility functions. It can be accessed by importing it from the radish. The world object is a threadlocal object so it is safe to use in threads.

You should not be using world to store data in scenarios and steps, that is what Contexts are for.

The config attribute of world world contains a Configuration object with named and positional arguments passed into radish. A basic transformation is applied to each of the arguments to turn it into a python attribute: As such “-” is replaced with “_”, “–” is removed, and “<” and “>” characters are removed.

For example --bdd-xml argument can be accessed using world.config.bdd_xml, and the argument <features> are accesses as world.config.features.

# -*- coding: utf-8 -*-

from radish import world

# print basedir
print(world.config.basedir)

# print profile
print(worl.config.profile)

Sometimes it’s useful to have specific variables and functions available during a whole test run. These variables and functions can be added to the world object:

# -*- coding: utf-8 -*-

from radish import world, pick
import random

world.x = 42

@pick
def get_magic_number():
    return random.randint(1, world.x)

The pick decorator adds the decorated function to the world object. You can use this function later in a step implementation or another hook:

# -*- coding: utf-8 -*-

from radish import before, world

from security import Tokenizer

@before.each_scenario
def gen_token(scenario):
   scenario.context.token = Tokenizer(world.get_magic_number())

BDD XML Report

Radish can report in the BDD XML format using --bdb-xml. The format of the XML is defined as follows:

XML declaration

<?xml version='1.0' encoding='utf-8'?>

<testrun> is a top level tag

agent:Agent of the test run composed of the user and hostname of the machine. Format: user@hostname
duration:Duration of test run in seconds rounded to the 10 decimal points.
starttime:Start time of the testrun run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS
endtime:End time of the testrun run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS

example:

<testrun>
  agent="user@computer"
  duration="0.0005660000"
  starttime="2017-02-18T07:06:55">
  endtime="2017-02-18T07:06:56"
>

The <testrun> contains the following tags

<feature> tag

id:Test run index id of the Feature. First feature to run is 1, second is 2 and so on.
sentence:Feature sentence.
result:Run state result of Feature run as described in Run state result
testfile:Path to the file name containing the feature. The path is relative to the basedir.
duration:Duration of Feature run in seconds rounded to the 10 decimal points.
starttime:Start time of the Feature run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS
endtime:End time of the Feature run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS

example:

<feature
  id="1"
  sentence="Step Parameters (tutorial03)"
  result="failed"
  testfile="./example.feature"
  duration="0.0008730000"
  starttime="2017-02-18T07:06:55"
  endtime="2017-02-18T07:06:55"
>

The <feature> tag contains the following tags:

<description> tag:

tag content:CDATA enclosed description of the feature.
<description>
  <![CDATA[This feature test following functionality
  - awesomeness
  - more awesomeness
  ]]>
</description>

<scenarios> tag:

Contains list of <screnario> tags

example:

<scenarios>

The <scenarios> tag contains the following tags:

<scenario> tag:

id:Test run index id of the Scenario. First scenario to run is 1, second is 2 and so on.
sentence:Scenario sentence.
result:Run state result of Scenario run as described in Run state result
testfile:Path to the file name containing the Scenario. The path is relative to the basedir.
duration:Duration of Scenario run in seconds rounded to the 10 decimal points.
starttime:Start time of the Scenario run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS
endtime:End time of the Scenario run. Combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS

example:

<scenario
  id="1"
  sentence="Blenders"
  result="failed"
  testfile="./example.feature"
  duration="0.0007430000"
  endtime="2017-02-18T07:06:55"
  starttime="2017-02-18T07:06:55"
>

The <scenario> tag contains the following tags:

<step> tag:

id:Test run index id of the Step. First Step to run is 1, second is 2 and so on.
sentence:Step sentence.
result:Run state result of Step run as described in Run state result
testfile:Path to the file name containing the Step. The path is relative to the basedir.
duration:Duration of Step run in seconds rounded to the 10 decimal points.
starttime:Start time of the Step run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS
endtime:End time of the Step run. Format: combined date and time representations, where date and time is separated by letter “T”. Format: YYYY-MM-DDTHH:MM:SS

example:

<step
  id="1"
  sentence="Given I put &quot;apples&quot; in a blender"
  result="passed"
  testfile="./example.feature"
  duration="0.0007430000"
  endtime="2017-02-18T07:06:55"
  starttime="2017-02-18T07:06:55"
>

The <step> MAY tag contains the following tags if error has occured:

<failure> tag:

message:Test run index id of the Step. First Step to run is 1, second is 2 and so on.
type:Step sentence.
tag content:CDATA enclosed failure reason specifically excepion traceback.

example:

<failure message="hello" type="Exception">
   <![CDATA[Traceback (most recent call last):
     File "/tmp/bdd/_env36/lib/python3.6/site-packages/radish/stepmodel.py", line 91, in run
         self.definition_func(self, \*self.arguments)  # pylint: disable=not-callable
     File "/tmp/bdd/radish/radish/example.py", line 34, in step_when_switch_blender_on
       raise Exception("show off radish error handling")
   Exception: show off radish error handling
  ]]>
</failure>

Testing Step Patterns

New since radish version v0.3.0

Radish provides a nice way to test if the implemented step pattern (@step(...)) match the expected sentences. This is especially useful if you provide a set of step implementations and someone else is going to use them and implement the feature files.

In a way your step pattern are the interface of your step implementation and interfaces ought to be tested properly.

If you’ve installed radish a command called radish-test is available. Install it’s dependencies with:

pip install radish-bdd[testing]

The matches sub command is used to test your step pattern inside your base dirs (-b / --basedir) against some sentences defined in a YAML file. We call those files match configs. A match config file has the following format:

- sentence: <SOME STEP SENTENCE>
  should_match: <THE STEP FUNCTION NAME IT SHOULD MATCH>
  with-arguments:
    # argument check if implicit type
    - <ARGUMENT 1 NAME>: <ARGUMENT 1 VALUE>
    # argument check with explicit type
    - <ARGUMENT 2 NAME>:
        type: <ARGUMENT 2 TYPE NAME>
        value: <ARGUMENT 2 VALUE>
sentence:Required. This is the sentence you want to test. It’s an example of a sentence which should match a certain Step pattern.
should_match:Required. This is the name of the python Step implementation function which you expect the sentence will match with.
with-arguments:Optional. This is a list of arguments which you expect will be passed in the python Step implementation function. The arguments can be specified as key-value pairs or as an object with a type and value. This could be useful if a custom argument expression is used to parse the arguments.

Example

Let’s assume we have the following step.py implementation:

from radish.stepregistry import step
from radish import given, when, then

@step("I have the number {number:g}")
def have_number(step, number):
    step.context.numbers.append(number)


@when("I sum them")
def sum_numbers(step):
    step.context.result = sum(step.context.numbers)


@then("I expect the result to be {result:g}")
def expect_result(step, result):
    assert step.context.result == result

And a step-matches.yml file like this:

- sentence: Given I have the number 5
  should_match: have_number
  with-arguments:
      - number:
            type: float
            value: 5.0

- sentence: When I sum them
  should_match: sum_numbers

- sentence: Then I expect the result to be 8
  should_match: expect_result
  with-arguments:
      - result: 8.0

We can check the step.py implementation against the step-matches.yml match config file using the radish-test CLI application:

radish-test matches tests/step-matches.yml

Due to the fact that the step.py module is located in $PWD/radish we don’t have to specify it’s location with -b or --basedir.

For the radish-test call above we would get the following output:

Testing sentences from tests/step-matches.yml:
>> STEP "Given I have the number 5" SHOULD MATCH have_number ✔
>> STEP "When I sum them" SHOULD MATCH sum_numbers ✔
>> STEP "Then I expect the result to be 8" SHOULD MATCH expect_result ✔

3 sentences (3 passed)
Covered 3 of 3 step implementations

In case of success we get the exit code 0 and in case of failure we’d get an exit code which is greater than 0.

radish-test matches also supports step coverage measurements. Use --cover-min-percentage to let radish-test matches fail if a certain coverage threshold is not met and use the --cover-show-missing command line option to list all not covered steps and their location.