dispel4py.storm package

Submodules

dispel4py.storm.client module

class dispel4py.storm.client.StormClient(host, port)

A simple Storm client that connects to the cluster at ‘host’:’port’.

killTopology(name)

Kills the topology identified by ‘name’.

submitTopology(name, uploaded_jar_location, conf, topology)

Submits the specified topology with name ‘name’ to the configured host.

Parameters:

• name – name given to the topology
• uploaded_jar_location – the path to the location of the jar as created by Storm
• conf – configuration of the topology
• topology – a Storm topology

dispel4py.storm.storm_submission module

Submits a dispel4py graph for processing to Storm. All dependencies must be available in the named resources directory or from the registry.

From the commandline, run the following:

dispel4py storm <module> [name] [-h] -m {local,remote,create}
                         [-r resourceDir] [-a attribute] [-s]

with positional arguments:

module:module that creates a dispel4py graph name:name of Storm topology to submit (optional)

and optional arguments:

-h, --help	show this help message and exit
-m, --mode mode
	execution mode, one of {local, remote, create}
-r, --resources resourceDir
	path to local modules used by the graph - default ”./resources/” (optional)
-a, --attr attribute
	name of graph variable in the module (optional)
-s, --save	do not remove Storm resources after submission (default is to remove resources)

Note

A Storm topology, once submitted, runs forever until it is explicitly killed.

dispel4py.storm.storm_submission.create(workflow, args, inputs)

Creates a Storm submission package and prints the temp directory containing the package.

Parameters:

• mod – module that creates a dispel4py graph
• attr – name of graph attribute within the module - if None

the first WorkflowGraph is used :param res: resource directory

dispel4py.storm.storm_submission.createPackage(graph, args, static_input)

Creates a Storm submission package for the given dispel4py graph.

Parameters:

• module_name – name of the graph module that creates a graph
• attr – name of the graph attribute within the module - if None the

first WorkflowGraph is used :param res: resource directory - if None the default is “resources” :rtype: name of the temporary directory that contains the submission package

dispel4py.storm.storm_submission.parse_args(args, namespace)

dispel4py.storm.storm_submission.process(workflow, inputs, args=None)

dispel4py.storm.storm_submission.runLocal(workflow, args, inputs)

Creates a Storm submission package and executes it locally. Note that the Storm topology runs until the process is explicitly killed, for example by pressing Ctrl-C.

Parameters:

• mod – module that creates a dispel4py graph
• attr – name of graph attribute within the module - if None the first

WorkflowGraph is used :param res: resource directory :param save: if True the Storm submission package is not deleted at the end of the run

dispel4py.storm.storm_submission.submit(workflow, args, inputs)

Creates a Storm submission package and submits it to a remote cluster.

Parameters:

• mod – module that creates a dispel4py graph
• attr – name of graph attribute within the module - if None the first

WorkflowGraph is used :param res: resource directory :param save: if True the Storm submission package is not deleted at the end of the run

dispel4py.storm.storm_submission_client module

dispel4py.storm.topology module

Creates a Storm topology from a dispel4py graph.

dispel4py.storm.topology.buildTopology(workflow)

Builds a Storm topology from the given dispel4py workflow.

dispel4py.storm.utils module

dispel4py.storm.utils.stormLogger(message)

Logger implementation for Storm.

Module contents

Tools for the enactment of Dispel4Py graphs as Storm topologies.