Reference: dispel4py modulesΒΆ
Core modules
dispel4py.core |
The core module for dispel4py. |
dispel4py.base |
Base PEs implementing typical processing patterns. |
dispel4py.utils |
Collection of dispel4py utilities. |
dispel4py.visualisation |
The IPython module for visualising a dispel4py graph using Graphviz dot. |
dispel4py.workflow_graph |
The dispel4py workflow graph. |
Enactment platforms
dispel4py.new.simple_process |
Simple sequential processor mapping for dispel4py graphs. |
dispel4py.new.multi_process |
Enactment of dispel4py graphs using multiprocessing. |
dispel4py.new.mpi_process |
Enactment of dispel4py graphs with MPI. |
dispel4py.new.processor |
Enactment of dispel4py graphs. |
dispel4py.storm.storm_submission |
Submits a dispel4py graph for processing to Storm. |
dispel4py.storm.topology |
Creates a Storm topology from a dispel4py graph. |
Examples and tests
dispel4py.examples.graph_testing.pipeline_test |
This is a dispel4py graph which produces a pipeline workflow with one producer node (prod) and 5 consumer nodes. |
dispel4py.examples.graph_testing.split_merge |
This is a dispel4py graph which produces a workflow that splits the data and sends it to two nodes (cons1 and cons2) and the output of those two nodes is merged by another node (last). |
dispel4py.examples.graph_testing.teecopy |
This is a dispel4py graph which produces a workflow that copies the data (from node prod) to two nodes (cons1 and cons2). |
dispel4py.examples.graph_testing.group_by |
This is a dispel4py graph that shows the group-by data pattern to count words. |
dispel4py.examples.graph_testing.grouping_alltoone |
This is a dispel4py graph which produces a workflow with a pipeline in which the producer node prod sends data to the consumer node cons1 which then sends data to node cons2 . |
dispel4py.examples.graph_testing.grouping_onetoall |
This is a dispel4py graph which produces a workflow which copies the data (from node prod ) to one node (cons ). |
dispel4py.examples.graph_testing.grouping_split_merge |
This is a dispel4py graph which produces a workflow that sends copies of the output data from the producer node (words) to two nodes (filter1 and filter2), and the outputs of those two filters are merged in the last node (count). |
dispel4py.examples.graph_testing.parallel_pipeline |
This is a dispel4py graph where each MPI process computes a partition of the workflow instead of a PE instance. |
dispel4py.examples.graph_testing.partition_parallel_pipeline |
This graph is a modification of the parallel_pipeline example, showing how the user can specify how the graph is going to be partitioned into MPI processes. |
dispel4py.examples.graph_testing.pipeline_composite |
This is a dispel4py graph which produces a pipeline workflow with one producer node and a chain of functions that process the data. |
dispel4py.examples.graph_testing.word_count |
Counts words produced by a WordProducer. |
dispel4py.examples.graph_testing.word_count_filter |
Counts words produced by RandomWordProducer and filtered by RandomFilter. |
dispel4py.examples.graph_testing.testing_PEs |
Example PEs for test workflows, implementing various patterns. |
dispel4py.test.simple_process_test |
Tests for simple sequential processing engine. |
dispel4py.new.aggregate |
Processing elements that implement aggregation functions (AVG, SUM, COUNT, MIN, MAX). |
dispel4py.new.aggregate_test |
Tests for aggregation processing elements. |