dispel4py.new package

Submodules

dispel4py.new.aggregate module

Processing elements that implement aggregation functions (AVG, SUM, COUNT, MIN, MAX). These are composite PEs that are automatically parallelised if the mapping supports this.

class dispel4py.new.aggregate.AggregatePE(indexes=[0])

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

class dispel4py.new.aggregate.AverageParallelPE(index=0)

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

class dispel4py.new.aggregate.AverageReducePE

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

class dispel4py.new.aggregate.ContinuousReducePE(indexes=[0])

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

process(inputs)

class dispel4py.new.aggregate.CountPE

Bases: dispel4py.new.aggregate.AggregatePE

class dispel4py.new.aggregate.MaxPE(indexes=[0])

Bases: dispel4py.new.aggregate.AggregatePE

class dispel4py.new.aggregate.MinPE(indexes=[0])

Bases: dispel4py.new.aggregate.AggregatePE

class dispel4py.new.aggregate.StdDevPE(index=0)

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

class dispel4py.new.aggregate.StdDevReducePE

Bases: dispel4py.core.GenericPE

INPUT_NAME = 'input'

OUTPUT_NAME = 'output'

class dispel4py.new.aggregate.SumPE(indexes=[0])

Bases: dispel4py.new.aggregate.AggregatePE

dispel4py.new.aggregate.parallelAvg(index=0)

Creates an AVG composite PE that can be parallelised using a map-reduce pattern.

dispel4py.new.aggregate.parallelCount()

Creates a counter composite PE that is parallelisable using a map-reduce pattern. The first part of the composite PE is a counter that counts all the inputs, the second part sums up the counts of the counter instances. The output of this PE is a single value that is the number of input items.

dispel4py.new.aggregate.parallelMax(indexes=[0])

Creates a MAX composite PE that can be parallelised using a map-reduce pattern.

dispel4py.new.aggregate.parallelMin(indexes=[0])

Creates a MIN composite PE that can be parallelised using a map-reduce pattern.

dispel4py.new.aggregate.parallelStdDev(index=0)

Creates a STDDEV composite PE that can be parallelised using a map-reduce pattern.

dispel4py.new.aggregate.parallelSum(indexes=[0])

Creates a SUM composite PE that can be parallelised using a map-reduce pattern.

dispel4py.new.aggregate.parallel_aggregate(instPE, reducePE)

dispel4py.new.aggregate_test module

Tests for aggregation processing elements.

Using nose (https://nose.readthedocs.org/en/latest/) run as follows:

nosetests dispel4py/new/aggregate_test.py
....
----------------------------------------------------------------------
Ran 4 tests in 0.023s

OK

dispel4py.new.aggregate_test.graph_avg()

dispel4py.new.aggregate_test.graph_count()

dispel4py.new.aggregate_test.graph_min_max()

dispel4py.new.aggregate_test.graph_sum()

dispel4py.new.aggregate_test.testAvg()

dispel4py.new.aggregate_test.testCount()

dispel4py.new.aggregate_test.testMinMax()

dispel4py.new.aggregate_test.testSum()

dispel4py.new.mappings module

dispel4py.new.monitoring module

class dispel4py.new.monitoring.MonitoringWrapper(baseObject)

Bases: object

class dispel4py.new.monitoring.ProcessTimingPE(baseObject)

Bases: dispel4py.new.monitoring.MonitoringWrapper

process(inputs)

class dispel4py.new.monitoring.ReadTimingWrapper(baseObject)

Bases: dispel4py.new.monitoring.MonitoringWrapper

class dispel4py.new.monitoring.Timer(verbose=False)

Bases: object

dispel4py.new.mpi_process module

Enactment of dispel4py graphs with MPI.

From the commandline, run the following command:

dispel4py mpi <module> [-h] [-a attribute] [-f inputfile] [-i iterations]

with parameters

module:module that creates a Dispel4Py graph -a attr:name of the graph attribute within the module (optional) -f file:file containing input data in JSON format (optional) -i iter:number of iterations to compute (default is 1) -h:print this help page

For example:

mpiexec -n 6 dispel4py mpi dispel4py.examples.graph_testing.pipeline_test        -i 5
Processing 5 iterations.
Processing 5 iterations.
Processing 5 iterations.
Processing 5 iterations.
Processing 5 iterations.
Processing 5 iterations.
Processes: {'TestProducer0': [5], 'TestOneInOneOut5': [2],        'TestOneInOneOut4': [4], 'TestOneInOneOut3': [3],        'TestOneInOneOut2': [1], 'TestOneInOneOut1': [0]}
TestOneInOneOut1 (rank 0): Processed 5 iterations.
TestOneInOneOut2 (rank 1): Processed 5 iterations.
TestOneInOneOut3 (rank 3): Processed 5 iterations.
TestProducer0 (rank 5): Processed 5 iterations.
TestOneInOneOut4 (rank 4): Processed 5 iterations.
TestOneInOneOut5 (rank 2): Processed 5 iterations.

class dispel4py.new.mpi_process.MPIWrapper(pe, provided_inputs=None)

Bases: dispel4py.new.processor.GenericWrapper

dispel4py.new.mpi_process.main()

dispel4py.new.mpi_process.parse_args(args, namespace)

dispel4py.new.mpi_process.process(workflow, inputs, args)

dispel4py.new.multi_process module

Enactment of dispel4py graphs using multiprocessing.

From the commandline, run the following command:

dispel4py multi <module> -n num_processes [-h] [-a attribute]                    [-f inputfile] [-i iterations]

with parameters

module:module that creates a Dispel4Py graph -n num:number of processes (required) -a attr:name of the graph attribute within the module (optional) -f file:file containing input data in JSON format (optional) -i iter:number of iterations to compute (default is 1) -h:print this help page

For example:

dispel4py multi dispel4py.examples.graph_testing.pipeline_test -i 5 -n 6
Processing 5 iterations.
Processes: {'TestProducer0': [5], 'TestOneInOneOut5': [2],                'TestOneInOneOut4': [4], 'TestOneInOneOut3': [3],                'TestOneInOneOut2': [1], 'TestOneInOneOut1': [0]}
TestProducer0 (rank 5): Processed 5 iterations.
TestOneInOneOut1 (rank 0): Processed 5 iterations.
TestOneInOneOut2 (rank 1): Processed 5 iterations.
TestOneInOneOut3 (rank 3): Processed 5 iterations.
TestOneInOneOut4 (rank 4): Processed 5 iterations.
TestOneInOneOut5 (rank 2): Processed 5 iterations.

class dispel4py.new.multi_process.MultiProcessingWrapper(rank, pe, provided_inputs=None)

Bases: dispel4py.new.processor.GenericWrapper

dispel4py.new.multi_process.main()

dispel4py.new.multi_process.parse_args(args, namespace)

dispel4py.new.multi_process.process(workflow, inputs, args)

dispel4py.new.processor module

Enactment of dispel4py graphs. This module contains methods that are used by different mappings.

From the commandline, run the following command:

dispel4py <mapping> <module>  [-h] [-a attribute] [-f inputfile]                                  [-i iterations] [...]

with parameters

mapping:target mapping module:module that creates a Dispel4Py graph -n num:number of processes (required) -a attr:name of the graph attribute within the module (optional) -f file:file containing input data in JSON format (optional) -i iter:number of iterations to compute (default is 1) -h:print this help page

Other parameters might be required by the target mapping, for example the number of processes if running in a parallel environment.

class dispel4py.new.processor.AllToOneCommunication(destinations)

Bases: object

getDestination(data)

class dispel4py.new.processor.GenericWrapper(pe)

Bases: object

process()

sources

class dispel4py.new.processor.GenericWriter(wrapper, name)

Bases: object

write(data)

class dispel4py.new.processor.GroupByCommunication(destinations, input_name, groupby)

Bases: object

getDestination(data)

class dispel4py.new.processor.OneToAllCommunication(destinations)

Bases: object

getDestination(data)

class dispel4py.new.processor.ShuffleCommunication(rank, sources, destinations)

Bases: object

getDestination(data)

class dispel4py.new.processor.SimpleProcessingPE(input_mappings, output_mappings, proc_to_pe)

Bases: dispel4py.core.GenericPE

A PE that processes a subgraph of PEs in sequence.

class dispel4py.new.processor.SimpleWriter(simple_pe, pe, output_mappings, result_mappings=None)

Bases: object

write(output_name, data)

dispel4py.new.processor.assign_and_connect(workflow, size)

dispel4py.new.processor.create_arg_parser()

dispel4py.new.processor.create_inputs(args, graph)

dispel4py.new.processor.create_partitioned(workflow_all)

dispel4py.new.processor.get_inputs(pe, inputs)

dispel4py.new.processor.get_partitions(workflow)

dispel4py.new.processor.load_graph_and_inputs(args)

dispel4py.new.processor.main()

dispel4py.new.processor.map_inputs_to_partitions(ubergraph, inputs)

dispel4py.new.processor.parse_common_args()

dispel4py.new.processor.simpleLogger(self, msg)

dispel4py.new.simple_process module

Simple sequential processor mapping for dispel4py graphs. This processor determines the dependencies of each PE in the graph and executes them sequentially.

From the commandline, run the following command:

dispel4py simple <module|module file> \
          [-a attribute] \
          [-f inputfile] \
          [-d inputdata] \
          [-i iterations] \
          [-h]

with parameters

module:module that creates a Dispel4Py graph -a attr:name of the graph attribute within the module (optional) -f file:file containing input data in JSON format (optional) -d data:input data in JSON format (optional) -i iter:number of iterations to compute (default is 1) -h:print this help page

The input data must be a dictionary mapping either a PE name or an PE identifier to a list of input data or the number of iterations which is a non-negative integer.

For example:

dispel4py simple dispel4py.examples.graph_testing.pipeline_test -i 5

Processing 5 iterations.
Inputs: {'TestProducer0': 5}
SimplePE: Processed 1 iteration.
Outputs: {'TestOneInOneOut5': {'output': [1, 2, 3, 4, 5]}}

The simple processor can also be called from an interactive Python session. For example, the following source creates a simple graph with a producer and a consumer PE and executes 5 iterations:

from dispel4py.workflow_graph import WorkflowGraph
from dispel4py.examples.graph_testing.testing_PEs \
    import TestProducer, TestOneInOneOut

graph = WorkflowGraph()
prod = TestProducer()
cons = TestOneInOneOut()
graph.connect(prod, 'output', cons, 'input')

from dispel4py.new.simple_process import process_and_return
results = process_and_return(graph, { prod : 5 })

The return value is the collection the output data of the consumer PE as this output stream is not connected:

print results
{'TestOneInOneOut1': {'output': [1, 2, 3, 4, 5]}}

class dispel4py.new.simple_process.SimpleProcessingWrapper(pe, provided_inputs=None)

Bases: dispel4py.new.processor.GenericWrapper

dispel4py.new.simple_process.main()

dispel4py.new.simple_process.process(workflow, inputs, args=None, resultmappings=None)

Executes the simple sequential processor for dispel4py graphs and prints out the input and output data. This is the default target when invoking the simple mapping with dispel4py simple <module>.

Parameters:

• workflow – the dispel4py graph to be enacted
• inputs – inputs for root PEs of the graphs. This is a dictionary mapping a PE id to either a non-negative integer (the number of iterations) or a list of input data items.

dispel4py.new.simple_process.process_and_return(workflow, inputs, resultmappings=None)

Executes the simple sequential processor for dispel4py graphs and returns the data collected from any unconnected output streams.

Parameters:

• workflow – the dispel4py graph to be enacted
• inputs – inputs for root PEs of the graphs. This is a dictionary mapping a PE to either a non-negative integer (the number of iterations) or a list of input data items.

Return type:

a dictionary mapping PE ids to the output data produced by that PE

dispel4py.new.simple_process.simpleLogger(self, msg)

Module contents