dispel4py.examples.graph_testing package

Submodules

dispel4py.examples.graph_testing.delayed_pipeline module

This is a dispel4py graph which produces a pipeline workflow with one producer node (prod) and 2 consumer nodes. The second consumer node delays the output by a fixed time and records the average processing time.

Execution:

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> python -m dispel4py.worker_mpi\
           [-a name_dispel4py_graph]\
           [-f file containing the input dataset in JSON format]\
           [-i number of iterations/runs']\
           [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 6 dispel4py mpi \
        dispel4py.examples.graph_testing.delayed_pipeline
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 3 nodes. For this reason we need at least 3 MPI processes to execute it.

    Output:

    Processes: {'TestDelayOneInOneOut2': [2, 3], 'TestProducer0': [4], 'TestOneInOneOut1': [0, 1]}
    TestProducer0 (rank 4): Processed 10 iterations.
    TestOneInOneOut1 (rank 1): Processed 5 iterations.
    TestOneInOneOut1 (rank 0): Processed 5 iterations.
    TestDelayOneInOneOut2 (rank 3): Average processing time: 1.00058307648
    TestDelayOneInOneOut2 (rank 3): Processed 5 iterations.
    TestDelayOneInOneOut2 (rank 2): Average processing time: 1.00079641342
    TestDelayOneInOneOut2 (rank 2): Processed 5 iterations.
    
dispel4py.examples.graph_testing.delayed_pipeline.cons1 = <dispel4py.examples.graph_testing.testing_PEs.TestOneInOneOut object>

adding a processing timer

dispel4py.examples.graph_testing.delayed_pipeline.cons2 = <dispel4py.new.monitoring.TestDelayOneInOneOut object>

important: this is the graph_variable

dispel4py.examples.graph_testing.group_by module

This is a dispel4py graph that shows the group-by data pattern to count words.

dispel4py.examples.graph_testing.group_by.testGrouping()

Creates the test graph.

dispel4py.examples.graph_testing.grouping_alltoone module

This is a dispel4py graph which produces a workflow with a pipeline in which the producer node prod sends data to the consumer node cons1 which then sends data to node cons2. Note that in this graph we have defined several instances of the cons1 and cons2 nodes and all the instances of the cons1 node are sending data to only one instance of cons2 node. This type of grouping is called global in dispel4py (all to one).

_images/grouping_alltoone.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When <-i number of interations/runs> is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 11 dispel4py mpi \
        dispel4py.examples.graph_testing.grouping_alltoone -i 10
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 3 nodes. For this reason we need at least 3 MPI processes to execute it.

    Output:

    Processing 10 iterations.
    Processes: {'TestProducer0': [0], 'TestOneInOneOut2': [6, 7, 8, 9, 10], 'TestOneInOneOut1': [1, 2, 3, 4, 5]}
    TestProducer0 (rank 0): Processed 10 iterations.
    TestOneInOneOut1 (rank 1): Processed 2 iterations.
    TestOneInOneOut1 (rank 2): Processed 2 iterations.
    TestOneInOneOut1 (rank 4): Processed 2 iterations.
    TestOneInOneOut1 (rank 5): Processed 2 iterations.
    TestOneInOneOut1 (rank 3): Processed 2 iterations.
    TestOneInOneOut2 (rank 7): Processed 0 iterations.
    TestOneInOneOut2 (rank 8): Processed 0 iterations.
    TestOneInOneOut2 (rank 9): Processed 0 iterations.
    TestOneInOneOut2 (rank 6): Processed 10 iterations.
    TestOneInOneOut2 (rank 10): Processed 0 iterations.
    

    Note that only one instance of the consumer node TestOneInOneOut2 (rank 6) received all the input blocks from the previous PE, as the grouping has been defined as global.

  • STORM:

dispel4py.examples.graph_testing.grouping_alltoone.testAlltoOne()

Creates a graph with two consumer nodes and a global grouping.

Return type:the created graph

dispel4py.examples.graph_testing.grouping_onetoall module

This is a dispel4py graph which produces a workflow which copies the data (from node prod) to one node (cons).

Note, that in this graph, we have decided to assign two processes to instances of the same PE:

cons = t.TestOneInOneOut()
cons.numprocesses=2

Another interesting point in this example is how to define the different types of groupings. In this example we have:

cons.inputconnections['input']['grouping'] = 'all'

which means that the prod node sends copies of its output data to all the connected instances. For that reason, the node cons has to specify the input grouping ‘all’ to the connection. In grouping_split_merge, we use another type of grouping, which is group by. However, in that case, the grouping type is defined by the PE class (WordCounter) which means it applies to all instances of that class (unless it is explicitly overridden by an instance as we did above).

If you compare this graph with teecopy these look quite similar. However, they do different things. In this example, we have two instances of the same PE and the node prod sends the same data to both instances whereas in teecopy the node prod sends the same data to two different PEs.

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> python -m dispel4py.worker_mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When <-i number of interations/runs> is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 3 python -m dispel4py.worker_mpi\
        dispel4py.examples.graph_testing.grouping_onetoall
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 3 nodes. For this reason we need at least 3 MPI processes to execute it.

    Output:

    Processing 1 iterations
    Processes: {'TestProducer0': [2], 'TestOneInOneOut1': [0, 1]}
    TestOneInOneOut1 (rank 0): I'm a bolt
    TestOneInOneOut1 (rank 1): I'm a bolt
    TestProducer0 (rank 2): I'm a spout
    Rank 2: Sending terminate message to [0, 1]
    TestProducer0 (rank 2): Processed 1 input block(s)
    TestProducer0 (rank 2): Completed.
    TestOneInOneOut1 (rank 1): Processed 1 input block(s)
    TestOneInOneOut1 (rank 1): Completed.
    TestOneInOneOut1 (rank 0): Processed 1 input block(s)
    TestOneInOneOut1 (rank 0): Completed.
    
  • STORM:

dispel4py.examples.graph_testing.grouping_onetoall.testOnetoAll()

dispel4py.examples.graph_testing.grouping_split_merge module

This is a dispel4py graph which produces a workflow that sends copies of the output data from the producer node (words) to two nodes (filter1 and filter2), and the outputs of those two filters are merged in the last node (count).

_images/grouping_split_merge.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> python -m dispel4py.worker_mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When <-i number of interations/runs> is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 4 python -m dispel4py.worker_mpi\
        dispel4py.examples.graph_testing.grouping_split_merge
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 4 nodes. For this reason we need at least 4 MPI processes to execute it.

    Output:

    Processing 1 iterations
    Processes: {'RandomFilter2': [3], 'WordCounter3': [1], 'RandomFilter1': [0], 'RandomWordProducer0': [2]}
    RandomFilter1 (rank 0): I'm a bolt
    RandomWordProducer0 (rank 2): I'm a spout
    WordCounter3 (rank 1): I'm a bolt
    Rank 2: Sending terminate message to [0]
    Rank 2: Sending terminate message to [3]
    RandomWordProducer0 (rank 2): Processed 1 input block(s)
    RandomWordProducer0 (rank 2): Completed.
    RandomFilter2 (rank 3): I'm a bolt
    Rank 3: Sending terminate message to [1]
    RandomFilter2 (rank 3): Processed 1 input block(s)
    RandomFilter2 (rank 3): Completed.
    Rank 0: Sending terminate message to [1]
    RandomFilter1 (rank 0): Processed 1 input block(s)
    RandomFilter1 (rank 0): Completed.
    WordCounter3 (rank 1): Processed 2 input block(s)
    WordCounter3 (rank 1): Completed.
    

    Note

    As those PEs are filtering randomly the output could be completely different.

dispel4py.examples.graph_testing.grouping_split_merge.testGrouping()

Creates the test graph.

dispel4py.examples.graph_testing.large_teecopy module

This is a dispel4py graph which produces a workflow that copies the data (from node prod) to two nodes (cons2 and cons3). This example can be used to process a large number of data blocks for testing.

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs'] [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default. Where <number_of_blocks> is the number of blocks produced by the source PE in each iteration.

    For example:

    mpiexec -n 4 dispel4py mpi \
        dispel4py.examples.graph_testing.large_teecopy 1000000
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 4 nodes. For this reason we need at least 4 MPI processes to execute it.

    Output:

    Processing 1 iterations
    Processes: {'TestProducer0': [0], 'TestOneInOneOut2': [2], 'TestOneInOneOut1': [1]}
    TestProducer0 (rank 0): I'm a spout
    Rank 0: Sending terminate message to [1]
    Rank 0: Sending terminate message to [2]
    TestProducer0 (rank 0): Processed 1 input block(s)
    TestProducer0 (rank 0): Completed.
    TestOneInOneOut2 (rank 2): I'm a bolt
    TestOneInOneOut2 (rank 2): Processed 1 input block(s)
    TestOneInOneOut2 (rank 2): Completed.
    TestOneInOneOut1 (rank 1): I'm a bolt
    TestOneInOneOut1 (rank 1): Processed 1 input block(s)
    TestOneInOneOut1 (rank 1): Completed
    
dispel4py.examples.graph_testing.large_teecopy.testTee()

Creates a graph with two consumer nodes and a tee connection.

Return type:the created graph

dispel4py.examples.graph_testing.multi_producer module

dispel4py.examples.graph_testing.parallel_pipeline module

This is a dispel4py graph where each MPI process computes a partition of the workflow instead of a PE instance. This happens automatically when the graph has more nodes than MPI processes. In terms of internal execution, the user has control which parts of the graph are distributed to each MPI process. See partition_parallel_pipeline on how to specify the partitioning.

_images/parallel_pipeline.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When <-i number of interations/runs> is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 3 dispel4py mpi dispel4py.examples.parallel_pipeline -i 10
    

    Note

    To force the partitioning the graph must have more nodes than available MPI processes. This graph has 4 nodes and we use 3 MPI processes to execute it.

    Output:

    Processing 10 iterations
    Graph is too large for MPI job size: 4 > 3. Start simple processing.
    Partitions:  [TestProducer0], [TestOneInOneOut1, TestOneInOneOut2, TestOneInOneOut3]
    Processes: {'GraphWrapperPE5': [1, 2], 'GraphWrapperPE4': [0]}
    GraphWrapperPE4 (rank 0): I'm a spout
    GraphWrapperPE5 (rank 1): I'm a bolt
    Rank 0: Sending terminate message to [1, 2]
    GraphWrapperPE4 (rank 0): Processed 10 input block(s)
    GraphWrapperPE4 (rank 0): Completed.
    GraphWrapperPE5 (rank 1): Processed 5 input block(s)
    GraphWrapperPE5 (rank 1): Completed.
    GraphWrapperPE5 (rank 2): I'm a bolt
    GraphWrapperPE5 (rank 2): Processed 5 input block(s)
    GraphWrapperPE5 (rank 2): Completed.
    
dispel4py.examples.graph_testing.parallel_pipeline.testParallelPipeline()

Creates a graph with 4 nodes.

Return type:the created graph

dispel4py.examples.graph_testing.partition_parallel_pipeline module

This graph is a modification of the parallel_pipeline example, showing how the user can specify how the graph is going to be partitioned into MPI processes. In this example we are specifying that one MPI process is executing the pipeline of nodes prod, cons1 and cons2 and the other MPI processes are executing the remaining node cons3:

graph.partitions = [ [prod, cons1, cons2], [cons3] ]

It can be executed with MPI and Storm. Storm will ignore the partition information.

Execution:

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 3 python -m dispel4py.worker_mpi \
        dispel4py.examples.graph_testing.partition_parallel_pipeline -i 10
    

    Output:

    Partitions:  [TestProducer0, TestOneInOneOut1, TestOneInOneOut2], [TestOneInOneOut3]
    Processes: {'GraphWrapperPE5': [0, 1], 'GraphWrapperPE4': [2]}
    GraphWrapperPE5 (rank 0): I'm a bolt
    GraphWrapperPE5 (rank 1): I'm a bolt
    GraphWrapperPE4 (rank 2): I'm a spout
    Rank 2: Sending terminate message to [0, 1]
    GraphWrapperPE4 (rank 2): Processed 10 input block(s)
    GraphWrapperPE4 (rank 2): Completed.
    GraphWrapperPE5 (rank 1): Processed 5 input block(s)
    GraphWrapperPE5 (rank 1): Completed.
    GraphWrapperPE5 (rank 0): Processed 5 input block(s)
    GraphWrapperPE5 (rank 0): Completed.
    
dispel4py.examples.graph_testing.partition_parallel_pipeline.testParallelPipeline()

Creates the parallel pipeline graph with partitioning information.

Return type:the created graph

dispel4py.examples.graph_testing.pipeline_composite module

This is a dispel4py graph which produces a pipeline workflow with one producer node and a chain of functions that process the data.

Execution:

  • Simple processing:

    Execute the sequential mapping as follows:

    dispel4py simple dispel4py.examples.graph_testing.pipeline_composite\
        [-i number of iterations]
    

    By default, if the number of iterations is not specified, the graph is executed once.

    For example:

    dispel4py simple dispel4py.examples.graph_testing.pipeline_composite
    

    Output:

    Processing 1 iteration.
    Inputs: {'TestProducer4': 1}
    SimplePE: Processed 1 iteration.
    Outputs: {'PE_subtract3': {'output': [5]}}
    
  • MPI:

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 5 dispel4py mpi \
        dispel4py.examples.graph_testing.pipeline_composite
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 5 nodes (4 function PEs and one producer). For this reason we need at least 5 MPI processes to execute it.

    Output:

    Processing 1 iteration.
    Processes: {'PE_addTwo0': [2], 'TestProducer4': [0], 'PE_subtract3': [4], 'PE_multiplyByFour1': [1], 'PE_divideByTwo2': [3]}
    TestProducer4 (rank 0): Processed 1 iteration.
    PE_addTwo0 (rank 2): Processed 1 iteration.
    PE_multiplyByFour1 (rank 1): Processed 1 iteration.
    PE_divideByTwo2 (rank 3): Processed 1 iteration.
    PE_subtract3 (rank 4): Processed 1 iteration.
    
  • STORM:

    From the dispel4py directory launch the Storm submission client:

    dispel4py storm dispel4py.examples.graph_testing.pipeline_composite\
        -m remote
    
dispel4py.examples.graph_testing.pipeline_composite.addTwo(data)

Returns 2 + data.

dispel4py.examples.graph_testing.pipeline_composite.divideByTwo(data)

Returns data/2.

dispel4py.examples.graph_testing.pipeline_composite.multiplyByFour(data)

Returns 4 * data.

dispel4py.examples.graph_testing.pipeline_composite.subtract(data, n)

Returns data - n.

dispel4py.examples.graph_testing.pipeline_test module

This is a dispel4py graph which produces a pipeline workflow with one producer node (prod) and 5 consumer nodes. It can be executed with MPI and STORM.

_images/pipeline_test.png

Execution:

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 6 dispel4py mpi\
        dispel4py.examples.graph_testing.pipeline_test
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 6 nodes. For this reason we need at least 6 MPI processes to execute it.

    Output:

    Processing 10 iterations.
    Processes: {'TestProducer0': [5], 'TestOneInOneOut5': [2], 'TestOneInOneOut4': [4], 'TestOneInOneOut3': [3], 'TestOneInOneOut2': [1], 'TestOneInOneOut1': [0]}
    TestProducer0 (rank 5): Processed 10 iterations.
    TestOneInOneOut1 (rank 0): Processed 10 iterations.
    TestOneInOneOut2 (rank 1): Processed 10 iterations.
    TestOneInOneOut3 (rank 3): Processed 10 iterations.
    TestOneInOneOut4 (rank 4): Processed 10 iterations.
    TestOneInOneOut5 (rank 2): Processed 10 iterations.
    
  • STORM:

    From the dispel4py directory launch the Storm submission client:

    dispel4py storm dispel4py.examples.graph_testing.pipeline_test\
        -m remote
    

    Output:

    Spec'ing TestOneInOneOut1
    Spec'ing TestOneInOneOut2
    Spec'ing TestOneInOneOut3
    Spec'ing TestOneInOneOut4
    Spec'ing TestOneInOneOut5
    Spec'ing TestProducer6
    spouts {'TestProducer6': ... }
    bolts  {'TestOneInOneOut5': ... }
    Created Storm submission package in /var/folders/58/7bjr3s011kgdtm5lx58prc_40000gn/T/tmp5ePEq3
    Running: java -client -Dstorm.options= -Dstorm.home= ...
    Submitting topology 'TestTopology' to storm.example.com:6627 ...
    
dispel4py.examples.graph_testing.pipeline_test.testPipeline(graph)

Adds a pipeline to the given graph.

Return type:the created graph

dispel4py.examples.graph_testing.producer_tee module

dispel4py.examples.graph_testing.split_merge module

This is a dispel4py graph which produces a workflow that splits the data and sends it to two nodes (cons1 and cons2) and the output of those two nodes is merged by another node (last).

_images/split_merge.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi <module|module file>\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 4 dispel4py mpi dispel4py.examples.graph_testing.split_merge
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 4 nodes. For this reason we need at least 4 MPI processes to execute it.

    Output:

    Processing 1 iteration.
    Processes: {'TestProducer0': [1], 'TestOneInOneOutWriter2': [2], 'TestTwoInOneOut3': [0], 'TestOneInOneOut1': [3]}
    TestProducer0 (rank 1): Processed 1 iteration.
    TestOneInOneOut1 (rank 3): Processed 1 iteration.
    TestOneInOneOutWriter2 (rank 2): Processed 1 iteration.
    TestTwoInOneOut3 (rank 0): Processed 2 iterations.
    
dispel4py.examples.graph_testing.split_merge.testSplitMerge()

Creates the split/merge graph with 4 nodes.

Return type:the created graph

dispel4py.examples.graph_testing.teecopy module

This is a dispel4py graph which produces a workflow that copies the data (from node prod) to two nodes (cons1 and cons2). If you compare this graph with grouping_onetoall they look quite similar. However, they do different things. In this example, the nodes cons1 and cons2 are different PE and the node prod sends the same data to both PEs.

_images/teecopy.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi <module|module file>\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 3 dispel4py mpi dispel4py.examples.graph_testing.teecopy
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 3 nodes. For this reason we need at least 3 MPI processes to execute it.

    Output:

    Processing 1 iteration.
    Processes: {'TestProducer0': [0], 'TestOneInOneOut2': [2], 'TestOneInOneOut1': [1]}
    TestProducer0 (rank 0): Processed 1 iteration.
    TestOneInOneOut2 (rank 2): Processed 1 iteration.
    TestOneInOneOut1 (rank 1): Processed 1 iteration.
    
dispel4py.examples.graph_testing.teecopy.testTee()

Creates a graph with two consumer nodes and a tee connection.

Return type:the created graph

dispel4py.examples.graph_testing.testing_PEs module

Example PEs for test workflows, implementing various patterns.

class dispel4py.examples.graph_testing.testing_PEs.NumberProducer(numIterations=1)

Bases: dispel4py.core.GenericPE

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.ProvenanceLogger

Bases: dispel4py.core.GenericPE

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.RandomFilter

Bases: dispel4py.core.GenericPE

This PE randomly filters the input.

input_name = 'input'
output_name = 'output'
process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.RandomWordProducer

Bases: dispel4py.core.GenericPE

This PE produces a random word as an output.

process(inputs=None)
words = ['dispel4py', 'computing', 'mpi', 'processing', 'simple', 'analysis', 'data']
class dispel4py.examples.graph_testing.testing_PEs.TestDelayOneInOneOut(delay=1)

Bases: dispel4py.core.GenericPE

This PE outputs the input data.

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.TestMultiProducer(num_output=10)

Bases: dispel4py.core.GenericPE

class dispel4py.examples.graph_testing.testing_PEs.TestOneInOneOut

Bases: dispel4py.core.GenericPE

This PE outputs the input data.

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.TestOneInOneOutWriter

Bases: dispel4py.core.GenericPE

This PE copies the input to an output, but it uses the write method. Remember that the write function allows to produce more than one output block within one processing step.

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.TestProducer(numOutputs=1)

Bases: dispel4py.core.GenericPE

This PE produces a range of numbers

class dispel4py.examples.graph_testing.testing_PEs.TestTwoInOneOut

Bases: dispel4py.core.GenericPE

This PE takes two inputs and it merges the data into one output string.

process(inputs)
class dispel4py.examples.graph_testing.testing_PEs.WordCounter

Bases: dispel4py.core.GenericPE

This PE counts the number of times (counter) that it receives each word. And it produces as an output: the same word (the input) and its counter.

input_name = 'input'
output_name = 'output'

dispel4py.examples.graph_testing.unconnected_pipeline module

This is a dispel4py graph which produces two pipeline workflows which are unconnected.

_images/unconnected_pipeline.png

It can be executed with MPI and STORM.

  • MPI: Please, locate yourself into the dispel4py directory.

    Execute the MPI mapping as follows:

    mpiexec -n <number mpi_processes> dispel4py mpi <module|module file>\
        [-a name_dispel4py_graph]\
        [-f file containing the input dataset in JSON format]\
        [-i number of iterations/runs']\
        [-s]
    

    The argument ‘-s’ forces to run the graph in a simple processing, which means that the first node of the graph will be executed in a process, and the rest of nodes will be executed in a second process. When [-i number of interations/runs] is not indicated, the graph is executed once by default.

    For example:

    mpiexec -n 12 dispel4py mpi \
        dispel4py.examples.graph_testing.unconnected_pipeline
    

    Note

    Each node in the graph is executed as a separate MPI process. This graph has 12 nodes. For this reason we need at least 12 MPI processes to execute it.

    Output:

    Processing 1 iteration.
    Processes: {'TestProducer0': [9], 'TestProducer6': [6], 'TestOneInOneOut9': [7], 'TestOneInOneOut8': [8], 'TestOneInOneOut7': [11], 'TestOneInOneOut5': [1], 'TestOneInOneOut4': [2], 'TestOneInOneOut3': [4], 'TestOneInOneOut2': [0], 'TestOneInOneOut1': [10], 'TestOneInOneOut11': [3], 'TestOneInOneOut10': [5]}
    TestProducer6 (rank 6): Processed 1 iteration.
    TestProducer0 (rank 9): Processed 1 iteration.
    TestOneInOneOut1 (rank 10): Processed 1 iteration.
    TestOneInOneOut7 (rank 11): Processed 1 iteration.
    TestOneInOneOut2 (rank 0): Processed 1 iteration.
    TestOneInOneOut8 (rank 8): Processed 1 iteration.
    TestOneInOneOut9 (rank 7): Processed 1 iteration.
    TestOneInOneOut10 (rank 5): Processed 1 iteration.
    TestOneInOneOut3 (rank 4): Processed 1 iteration.
    TestOneInOneOut4 (rank 2): Processed 1 iteration.
    TestOneInOneOut11 (rank 3): Processed 1 iteration.
    TestOneInOneOut5 (rank 1): Processed 1 iteration.
    
dispel4py.examples.graph_testing.unconnected_pipeline.testPipeline(graph)

Creates a pipeline and adds it to the given graph.

Return type:the modified graph
dispel4py.examples.graph_testing.unconnected_pipeline.testUnconnected()

Creates a graph with two unconnected pipelines.

Return type:the created graph

dispel4py.examples.graph_testing.word_count module

Counts words produced by a WordProducer.

dispel4py.examples.graph_testing.word_count_filter module

Counts words produced by RandomWordProducer and filtered by RandomFilter.

Module contents

Examples of dispel4py graphs.