mrv.automation.workflow

Epydoc: mrv.automation.workflow

Contains workflow classes that conenct processes in a di - graph

Classes

Epydoc: mrv.automation.workflow

class mrv.automation.workflow.Graph(**kwargs)

Bases: networkx.classes.digraph.DiGraph, mrv.interface.iDuplicatable

Holds the nodes and their connections

Nodes are kept in a separate list whereas the plug connections are kept in the underlying DiGraph

addNode(node)

Add a new node instance to the graph :note: node membership is exclusive, thus node instances can only be in one graph at a time :return: self, for chained calls

classmethod addSep(item, sep)

Returns:

item with separator added to it ( just once )

Note:

operates best on strings

Parameters:

• item – item to add separator to
• sep – the separator

add_cycle(nlist, **attr)

add_edge(u, v, attr_dict=None, **attr)

add_edges_from(ebunch, attr_dict=None, **attr)

add_node(n, attr_dict=None, **attr)

add_nodes_from(nodes, **attr)

add_path(nlist, **attr)

add_star(nlist, **attr)

add_weighted_edges_from(ebunch, **attr)

adjacency_iter()

adjacency_list()

basename()

Returns:basename of this path, ‘/hello/world’ -> ‘world’

children(predicate=<function <lambda> at 0x7f41dd7c2ed8>)

Returns:list of intermediate children of path, [ child1 , child2 ] Parameter:predicate – return True to include x in result Note:the child objects returned are supposed to be valid paths, not just relative paths

childrenDeep(order=1, predicate=<function <lambda> at 0x7f41dd7c5050>)

Returns:

list of all children of path, [ child1 , child2 ]

Parameters:

• order – order enumeration
• predicate – returns true if x may be returned

Note:

the child objects returned are supposed to be valid paths, not just relative paths

clear()

clearCache()

Clear the cache of all nodes in the graph - this forces the graph to reevaluate on the next request

connect(sourceshell, destinationshell, force=False)

Connect this plug to destinationshell such that destinationshell is an input plug for our output

Parameters:

• sourceshell – PlugShell being source of the connection
• destinationshell – PlugShell being destination of the connection
• force – if False, existing connections to destinationshell will not be broken, but an exception is raised if True, existing connection may be broken

Returns:

self on success, allows chained connections

Raises PlugAlreadyConnected:  

if destinationshell is connected and force is False

Raises PlugIncompatible:  

if destinationshell does not appear to be compatible to this one

copy()

copyFrom(other)

Duplicate all data from other graph into this one, create a duplicate of the nodes as well

copyTo(instance, *args, **kwargs)

Copy the values of ourselves onto the given instance which must be an instance of our class to be compatible. Only the common classes will be copied to instance

Returns:altered instance Note:instance will be altered during the process

copyToOther(instance, *args, **kwargs)

As copyTo, but does only require the objects to have a common base. It will match the actually compatible base classes and call copyFrom if possible. As more checking is performed, this method performs worse than copyTo

createInstance()

Create a copy of self and return it

degree(nbunch=None, with_labels=False, weighted=False)

degree_iter(nbunch=None, weighted=False)

disconnect(sourceshell, destinationshell)

Remove the connection between sourceshell to destinationshell if they are connected :note: does not raise if no connection is present

duplicate(*args, **kwargs)

Implements a c-style copy constructor by creating a new instance of self and applying the copyFrom methods from base to all classes implementing the copyfrom method. Thus we will call the method directly on the class

Parameters:

• args – passed to copyFrom and createInstance method to give additional directions
• kwargs – see param args

edges(nbunch=None, data=False)

edges_iter(nbunch=None, data=False)

fullChildName(childname)

Add the given name to the string version of our instance :return: string with childname added like name _sep childname

get_edge_data(u, v, default=None)

hasNode(node)

Returns:True if the node is in this graph, false otherwise

has_edge(u, v)

has_node(n)

has_predecessor(u, v)

has_successor(u, v)

in_degree(nbunch=None, with_labels=False, weighted=False)

in_degree_iter(nbunch=None, weighted=False)

in_edges(nbunch=None, data=False)

in_edges_iter(nbunch=None, data=False)

input(plugshell)

Returns:the connected input plug of plugshell or None if there is no such connection Note:input plugs have on plug at most, output plugs can have more than one connected plug

isPartOf(other)

Returns:True if self is a part of other, and thus can be found in other Note:operates on strings only

isRoot()

Returns:True if this path is the root of the DAG

isRootOf(other)

Returns:True other starts with self Note:operates on strings Note:we assume other has the same type as self, thus the same separator

is_directed()

is_multigraph()

iterConnectedNodes(predicate=<function <lambda> at 0x2ed0aa0>)

Returns:generator returning all nodes that are connected in this graph, in no particular order. For an ordered itereration, use iterShells. Parameter:predicate – if True for node, it will be returned

iterNodes(predicate=<function <lambda> at 0x2ed09b0>)

Returns:generator returning all nodes in this graph Parameter:predicate – if True for node, it will be returned Note:there is no particular order

iterParents(predicate=<function <lambda> at 0x7f41dd7c5230>)

Returns:generator retrieving all parents up to the root Parameter:predicate – returns True for all x that you want to be returned

nbunch_iter(nbunch=None)

neighbors(n)

neighbors_iter(n)

nodeByID(nodeID)

Returns:instance of a node according to the given node id Raises NameError:  if no such node exists in graph

nodes()

Returns:immutable copy of the nodes used in the graph

nodes_iter(data=False)

nodes_with_selfloops()

numNodes()

Returns:number of nodes in the graph

number_of_edges(u=None, v=None)

number_of_nodes()

number_of_selfloops()

order()

out_degree(nbunch=None, with_labels=False, weighted=False)

out_degree_iter(nbunch=None, weighted=False)

out_edges(nbunch=None, data=False)

out_edges_iter(nbunch=None, data=False)

outputs(plugshell, predicate=<function <lambda> at 0x2ed0e60>)

Returns:a list of plugs being the destination of the connection to plugshell Parameter:predicate – plug will only be returned if predicate is true for it - shells will be passed in

parent()

Returns:parent of this path, ‘/hello/world’ -> ‘/hello’ or None if this path is the dag’s root

parentDeep()

Returns:all parents of this path, ‘/hello/my/world’ -> [ ‘/hello/my’,’/hello’ ]

predecessors(n)

predecessors_iter(n)

removeNode(node)

Remove the given node from the graph ( if it exists in it )

remove_edge(u, v)

remove_edges_from(ebunch)

remove_node(n)

remove_nodes_from(nbunch)

reverse(copy=True)

root()

Returns:the root of the DAG - it has no further parents

selfloop_edges(data=False)

size(weighted=False)

subgraph(nbunch, copy=True)

successors(n)

successors_iter(n)

supports(interface_type)

Returns:True if this instance supports the interface of the given type Parameter:interface_type – Type of the interface you require this instance to support Note:Must be used in case you only have a weak reference of your interface instance or proxy which is a case where the ordinary isinstance( obj, iInterface ) will not work

to_directed()

to_undirected()

writeDot(fileOrPath)

Write the connections in self to the given file object or path :todo: remove if no longer needed

Epydoc: mrv.automation.workflow.Workflow

class mrv.automation.workflow.Workflow(**kwargs)

Bases: mrv.dge.Graph

Implements a workflow as connected processes

Note:if you have to access the processes directly, use the DiGraph methods

class CallGraph

Bases: networkx.classes.digraph.DiGraph

Simple wrapper storing a call graph, keeping the root at which the call started

Note:this class is specialized to be used by workflows, its not general for that purpose

add_cycle(nlist, **attr)

add_edge(u, v, attr_dict=None, **attr)

add_edges_from(ebunch, attr_dict=None, **attr)

add_node(n, attr_dict=None, **attr)

add_nodes_from(nodes, **attr)

add_path(nlist, **attr)

add_star(nlist, **attr)

add_weighted_edges_from(ebunch, **attr)

adjacency_iter()

adjacency_list()

callRoot()

Returns:root at which the call started

callstackSize()

Returns:length of the callstack

clear()

copy()

degree(nbunch=None, with_labels=False, weighted=False)

degree_iter(nbunch=None, weighted=False)

edges(nbunch=None, data=False)

edges_iter(nbunch=None, data=False)

endCall(result)

End the call start started previously

Parameter:result – the result of the call

get_edge_data(u, v, default=None)

has_edge(u, v)

has_node(n)

has_predecessor(u, v)

has_successor(u, v)

in_degree(nbunch=None, with_labels=False, weighted=False)

in_degree_iter(nbunch=None, weighted=False)

in_edges(nbunch=None, data=False)

in_edges_iter(nbunch=None, data=False)

is_directed()

is_multigraph()

nbunch_iter(nbunch=None)

neighbors(n)

neighbors_iter(n)

nodes(data=False)

nodes_iter(data=False)

nodes_with_selfloops()

number_of_edges(u=None, v=None)

number_of_nodes()

number_of_selfloops()

order()

out_degree(nbunch=None, with_labels=False, weighted=False)

out_degree_iter(nbunch=None, weighted=False)

out_edges(nbunch=None, data=False)

out_edges_iter(nbunch=None, data=False)

predecessors(n)

predecessors_iter(n)

remove_edge(u, v)

remove_edges_from(ebunch)

remove_node(n)

remove_nodes_from(nbunch)

reverse(copy=True)

selfloop_edges(data=False)

size(weighted=False)

startCall(pdata)

Add a call of a process

subgraph(nbunch, copy=True)

successors(n)

successors_iter(n)

toCallList(reverse=True, pruneIfTrue=<function <lambda> at 0x3445f50>)

Returns:

flattened version of graph as list of ProcessData edges in call order , having the root as last element of the list

Parameters:

• pruneIfTrue – Function taking ProcessData to return true if the node should be pruned from the result
• reverse – if true, the calllist will be properly reversed ( taking childre into account

to_directed()

to_undirected()

class Workflow.ProcessData(process, plug, mode)

Bases: object

Allows to store additional information with each process called during the workflow

elapsed()

Returns:time to process the call

endtime

exception

index

mode

plug

process

result()

Returns:result stored in this instance, or None if it is not present or not alive

setResult(result)

Set the given result

Note:uses weak references as the tracking structure should not cause a possible mutation of the program flow ( as instances stay alive although code expects it to be deleted

starttime

Workflow.addNode(node)

Add a new node instance to the graph :note: node membership is exclusive, thus node instances can only be in one graph at a time :return: self, for chained calls

classmethod Workflow.addSep(item, sep)

Returns:

item with separator added to it ( just once )

Note:

operates best on strings

Parameters:

• item – item to add separator to
• sep – the separator

Workflow.add_cycle(nlist, **attr)

Workflow.add_edge(u, v, attr_dict=None, **attr)

Workflow.add_edges_from(ebunch, attr_dict=None, **attr)

Workflow.add_node(n, attr_dict=None, **attr)

Workflow.add_nodes_from(nodes, **attr)

Workflow.add_path(nlist, **attr)

Workflow.add_star(nlist, **attr)

Workflow.add_weighted_edges_from(ebunch, **attr)

Workflow.adjacency_iter()

Workflow.adjacency_list()

Workflow.basename()

Returns:basename of this path, ‘/hello/world’ -> ‘world’

Workflow.callgraph()

Returns:current callgraph instance Note:its strictly read-only and may not be changed

Workflow.children(predicate=<function <lambda> at 0x7f41dd7c2ed8>)

Returns:list of intermediate children of path, [ child1 , child2 ] Parameter:predicate – return True to include x in result Note:the child objects returned are supposed to be valid paths, not just relative paths

Workflow.childrenDeep(order=1, predicate=<function <lambda> at 0x7f41dd7c5050>)

Returns:

list of all children of path, [ child1 , child2 ]

Parameters:

• order – order enumeration
• predicate – returns true if x may be returned

Note:

the child objects returned are supposed to be valid paths, not just relative paths

Workflow.clear()

Workflow.clearCache()

Clear the cache of all nodes in the graph - this forces the graph to reevaluate on the next request

Workflow.connect(sourceshell, destinationshell, force=False)

Connect this plug to destinationshell such that destinationshell is an input plug for our output

Parameters:

• sourceshell – PlugShell being source of the connection
• destinationshell – PlugShell being destination of the connection
• force – if False, existing connections to destinationshell will not be broken, but an exception is raised if True, existing connection may be broken

Returns:

self on success, allows chained connections

Raises PlugAlreadyConnected:  

if destinationshell is connected and force is False

Raises PlugIncompatible:  

if destinationshell does not appear to be compatible to this one

Workflow.copy()

Workflow.copyFrom(other)

Only mode is required

Workflow.copyTo(instance, *args, **kwargs)

Copy the values of ourselves onto the given instance which must be an instance of our class to be compatible. Only the common classes will be copied to instance

Returns:altered instance Note:instance will be altered during the process

Workflow.copyToOther(instance, *args, **kwargs)

As copyTo, but does only require the objects to have a common base. It will match the actually compatible base classes and call copyFrom if possible. As more checking is performed, this method performs worse than copyTo

Workflow.createInstance()

Create a copy of self and return it

Workflow.createReportInstance(reportType)

Create a report instance that describes how the previous target was made

Parameter:reportType – Report to populate with information - it must be a Plan based class that can be instantiated and populated with call information. A report analyses the call dependency graph generated during dg evaluation and presents it. Returns:report instance whose makeReport method can be called to retrieve it

Workflow.degree(nbunch=None, with_labels=False, weighted=False)

Workflow.degree_iter(nbunch=None, weighted=False)

Workflow.disconnect(sourceshell, destinationshell)

Remove the connection between sourceshell to destinationshell if they are connected :note: does not raise if no connection is present

Workflow.duplicate(*args, **kwargs)

Implements a c-style copy constructor by creating a new instance of self and applying the copyFrom methods from base to all classes implementing the copyfrom method. Thus we will call the method directly on the class

Parameters:

• args – passed to copyFrom and createInstance method to give additional directions
• kwargs – see param args

Workflow.edges(nbunch=None, data=False)

Workflow.edges_iter(nbunch=None, data=False)

Workflow.fullChildName(childname)

Add the given name to the string version of our instance :return: string with childname added like name _sep childname

Workflow.get_edge_data(u, v, default=None)

Workflow.hasNode(node)

Returns:True if the node is in this graph, false otherwise

Workflow.has_edge(u, v)

Workflow.has_node(n)

Workflow.has_predecessor(u, v)

Workflow.has_successor(u, v)

Workflow.in_degree(nbunch=None, with_labels=False, weighted=False)

Workflow.in_degree_iter(nbunch=None, weighted=False)

Workflow.in_edges(nbunch=None, data=False)

Workflow.in_edges_iter(nbunch=None, data=False)

Workflow.input(plugshell)

Returns:the connected input plug of plugshell or None if there is no such connection Note:input plugs have on plug at most, output plugs can have more than one connected plug

Workflow.isPartOf(other)

Returns:True if self is a part of other, and thus can be found in other Note:operates on strings only

Workflow.isRoot()

Returns:True if this path is the root of the DAG

Workflow.isRootOf(other)

Returns:True other starts with self Note:operates on strings Note:we assume other has the same type as self, thus the same separator

Workflow.is_directed()

Workflow.is_multigraph()

Workflow.iterConnectedNodes(predicate=<function <lambda> at 0x2ed0aa0>)

Returns:generator returning all nodes that are connected in this graph, in no particular order. For an ordered itereration, use iterShells. Parameter:predicate – if True for node, it will be returned

Workflow.iterNodes(predicate=<function <lambda> at 0x2ed09b0>)

Returns:generator returning all nodes in this graph Parameter:predicate – if True for node, it will be returned Note:there is no particular order

Workflow.iterParents(predicate=<function <lambda> at 0x7f41dd7c5230>)

Returns:generator retrieving all parents up to the root Parameter:predicate – returns True for all x that you want to be returned

Workflow.makeDirtyReport(target, mode='single')

Returns:

list of tuple( shell, DirtyReport|None ) If a process ( shell.node ) is dirty, a dirty report will be given explaining why the process is dirty and needs an update

Parameters:

• target – target you which to check for it’s dirty state
• mode –
  • single - only the process assigned to evaluate target will be checked

  • multi - as single, but the whole callgraph will be checked, starting

    at the first node, stepping down the callgraph. This gives a per node dirty report.

  • deep - try to evaluate target, but fail if one process in the target’s

    call history is dirty

Workflow.makeTarget(target)

Parameter:target – target to make - can be class or instance Returns:result when producing the target

Workflow.makeTargets(targetList, errstream=None, donestream=None)

batch module compatible method allowing to make mutliple targets at once

Parameters:

• targetList – iterable providing the targets to make
• errstream – object with file interface allowing to log errors that occurred during operation
• donestream – if list, targets successfully done will be appended to it, if it is a stream, the string representation will be wrtten to it

Workflow.nbunch_iter(nbunch=None)

Workflow.neighbors(n)

Workflow.neighbors_iter(n)

Workflow.nodeByID(nodeID)

Returns:instance of a node according to the given node id Raises NameError:  if no such node exists in graph

Workflow.nodes()

Returns:immutable copy of the nodes used in the graph

Workflow.nodes_iter(data=False)

Workflow.nodes_with_selfloops()

Workflow.numNodes()

Returns:number of nodes in the graph

Workflow.number_of_edges(u=None, v=None)

Workflow.number_of_nodes()

Workflow.number_of_selfloops()

Workflow.order()

Workflow.out_degree(nbunch=None, with_labels=False, weighted=False)

Workflow.out_degree_iter(nbunch=None, weighted=False)

Workflow.out_edges(nbunch=None, data=False)

Workflow.out_edges_iter(nbunch=None, data=False)

Workflow.outputs(plugshell, predicate=<function <lambda> at 0x2ed0e60>)

Returns:a list of plugs being the destination of the connection to plugshell Parameter:predicate – plug will only be returned if predicate is true for it - shells will be passed in

Workflow.parent()

Returns:parent of this path, ‘/hello/world’ -> ‘/hello’ or None if this path is the dag’s root

Workflow.parentDeep()

Returns:all parents of this path, ‘/hello/my/world’ -> [ ‘/hello/my’,’/hello’ ]

Workflow.predecessors(n)

Workflow.predecessors_iter(n)

Workflow.removeNode(node)

Remove the given node from the graph ( if it exists in it )

Workflow.remove_edge(u, v)

Workflow.remove_edges_from(ebunch)

Workflow.remove_node(n)

Workflow.remove_nodes_from(nbunch)

Workflow.reverse(copy=True)

Workflow.root()

Returns:the root of the DAG - it has no further parents

Workflow.selfloop_edges(data=False)

Workflow.size(weighted=False)

Workflow.subgraph(nbunch, copy=True)

Workflow.successors(n)

Workflow.successors_iter(n)

Workflow.supports(interface_type)

Returns:True if this instance supports the interface of the given type Parameter:interface_type – Type of the interface you require this instance to support Note:Must be used in case you only have a weak reference of your interface instance or proxy which is a case where the ordinary isinstance( obj, iInterface ) will not work

Workflow.targetRating(target)

Returns:

int range(0,255) indicating how well a target can be made 0 means not at all, 255 means perfect.

Return value is tuple ( rate, PlugShell ), containing the process and plug with the highest rating or None if rate is 0

Walk the dependency graph such that leaf nodes have higher ratings than non-leaf nodes

Note:

you can use the process.ProcessBase enumeration for comparison

Workflow.targetSupportList()

Returns:list of all supported target type Note:this method is for informational purposes only

Workflow.to_directed()

Workflow.to_undirected()

Workflow.writeDot(fileOrPath)

Write the connections in self to the given file object or path :todo: remove if no longer needed

Exceptions

Epydoc: mrv.automation.workflow

class mrv.automation.workflow.ComputeError

Bases: exceptions.Exception

Thrown if the computation done by a plug failed by an unknown exception It will be passed on in the exception

args

message

Epydoc: mrv.automation.workflow.DirtyException

class mrv.automation.workflow.DirtyException(report='')

Bases: exceptions.Exception

Exception thrown when system is in dirty query mode and the process detects that it is dirty.

The exception can also contain a report that will be returned using the makeReport function.

Note:Thus exeption class must NOT be derived from ComputeError as it will be caught by the DG engine and mis-interpreted - unknown exceptions will just be passed on by it

args

makeReport()

Returns:printable report, usually a string or some object that responds to str() appropriately

message

report

Epydoc: mrv.automation.workflow.TargetError

class mrv.automation.workflow.TargetError

Bases: exceptions.ValueError

Thrown if target is now supported by the workflow ( and thus cannot be made )

args

message