Filters

Filter Levels

There are three points in the message flow at which a filter can act: before pre-parsing happens, after the pre-parse but before the main parse, and after the main parse.

Pre-parsing extracts a callsign from a string and uses it to look up the rest of the payload’s configuration, so pre-parse filters are specified per module and will act on everything that module receives. These are called pre-filters and are specified in the parser configuration document.

Intermediate filters act after a callsign has been found but before the message is parsed for data, so they can correct particular format errors a certain payload might be transmitting. Post-parse filters act after the data parsing has happened, so can tweak the output data. Both intermediate and post filters are specified in the payload section of a flight document.

Filter Syntax

Two types of filters are supported: normal and hotfix. Normal filters give a callable object, which must be in the Python path, and optionally may give a configuration object which will be passed as the second argument to the callable. Hotfix filters just supply some Python code, which is used as the body of a function given the incoming data as its sole argument. In either case, the filter must return the newly processed data.

Example of a normal filter:

{
    "type": "normal",
    "callable": "habitat.filters.upper_case"
}

# habitat/filters.py
def upper_case(data):
    return data.upper()

A normal filter with a configuration object:

{
    "type": "normal",
    "callable": "habitat.filters.daylight_savings",
    "config": {"time_field": 7}
}

# habitat/filters.py
def daylight_savings(data, config):
    time = data[config['time_field']]
    hour = int(time[0:2])
    data[config['time_field']] = str(hour + 1) + time[2:]
    return data

A hotfix filter:

{
    "type": "hotfix",
    "code": "parts = data.split(',')\nreturn '.'.join(parts)\n"
}

Which would be assembled into:

def f(data):
    parts = data.split(',')
    return '.'.join(parts)